From 40df22e95e00ecb48ed4ff03010e6966e7ad954f Mon Sep 17 00:00:00 2001 From: Dang Nguyen Date: Sun, 28 Jun 2026 17:13:18 -0500 Subject: [PATCH 1/5] Clean up the quality-proxy benchmark for public release Rename the "outcomes/conference" study to the quality-proxy study to match the paper, and rewrite both READMEs (top-level + conference_study) around the paper's four-proxy design, the real run flow, and reproducibility. Reproducibility: - The full 197-paper set regenerates deterministically via select_papers.py (default output moved from manifests/v1 to manifests/canonical/, written as full.json). The 74-paper frontier subset has no regeneration script and is not shipped; the README points readers to the authors for its manifest. - Fix the model roster everywhere to the paper's six backbones (four efficient + two frontier), dropping exploratory models. Pin models explicitly in baseline.yaml and a new frontier.yaml. - run_study.py and download_papers.py default to the canonical manifest. Usability: - coarse resolves its venv via COARSE_VENV_PYTHON instead of a hardcoded local path, with setup docs. - generate_report.py becomes the single results entry point: it discovers the (method, model) cells from a run's results and prints the paper's pairwise-accuracy tables with 95% CIs via ci_auc, so no hand-maintained table config is needed. - README leads with a cost preview and a smoke-test path; estimate_cost.py carries a TODO for the grok rate. Remove the duplicate benchmarks/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 4 +- README.md | 124 ++-- benchmarks/README.md | 88 --- benchmarks/conference_study/.gitignore | 1 + benchmarks/conference_study/README.md | 252 ++++---- .../conference_study/analyses/compute_auc.py | 4 +- .../analyses/generate_report.py | 592 +++--------------- .../analyses/report_scaleup.py | 4 +- .../competitors/coarse_adapter.py | 9 +- .../conference_study/configs/baseline.yaml | 17 +- .../conference_study/configs/coarse.yaml | 17 +- .../conference_study/configs/frontier.yaml | 26 + .../conference_study/download_papers.py | 8 +- benchmarks/conference_study/estimate_cost.py | 3 + benchmarks/conference_study/paper_metrics.py | 2 +- benchmarks/conference_study/run_study.py | 2 +- benchmarks/conference_study/select_papers.py | 24 +- 17 files changed, 362 insertions(+), 815 deletions(-) delete mode 100644 benchmarks/README.md create mode 100644 benchmarks/conference_study/configs/frontier.yaml diff --git a/.gitignore b/.gitignore index 034b5fd..ea7263f 100644 --- a/.gitignore +++ b/.gitignore @@ -25,7 +25,9 @@ review_results/ benchmarks/conference_study/results/ benchmarks/perturbation/perturbation_results/ -# conference_study study artifacts (not code) +# conference_study study artifacts (not code). The full paper set regenerates +# via select_papers.py, so manifests/ is ignored. The frontier subset has no +# regeneration script and is not shipped (request it from the authors). benchmarks/conference_study/manifests/ benchmarks/conference_study/reports/ # Symlink targets — trailing-slash patterns wouldn't match the symlinks diff --git a/README.md b/README.md index 4c6f61f..773c4e1 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # OpenAIReview -[![PyPI version](https://img.shields.io/pypi/v/openaireview.svg)](https://pypi.org/project/openaireview/) +[PyPI version](https://pypi.org/project/openaireview/) Our goal is provide thorough and detailed reviews to help researchers conduct the best research. See more examples [here](https://openaireview.github.io/). -![Example](assets/example.png) +Example ## Installation @@ -14,11 +14,13 @@ uv venv && uv pip install openaireview ``` For fast PDF processing (requires `MISTRAL_API_KEY`): + ```bash uv pip install "openaireview[mistral]" ``` For development: + ```bash git clone https://github.com/ChicagoHAI/OpenAIReview.git cd OpenAIReview @@ -42,12 +44,14 @@ uv venv && uv pip install -e . PDF extraction quality matters — math symbols, tables, and reading order all affect review quality. Four engines are supported, tried in order: -| Engine | Install | Best for | Notes | -|--------|---------|----------|-------| -| **Mistral OCR** | `pip install "openaireview[mistral]"` + set `MISTRAL_API_KEY` | Best overall quality, math, tables | Cloud API, ~$0.001/page | -| **DeepSeek OCR** | `pip install "openaireview[deepseek]"` + local backend | Privacy-sensitive docs | Local model via Ollama/vLLM | -| **Marker** | `uv tool install marker-pdf --with psutil` | Math-heavy PDFs (offline) | Slow without GPU | -| **pymupdf4llm** | (included) | Fallback, always available | No math symbol support | + +| Engine | Install | Best for | Notes | +| ---------------- | ------------------------------------------------------------- | ---------------------------------- | --------------------------- | +| **Mistral OCR** | `pip install "openaireview[mistral]"` + set `MISTRAL_API_KEY` | Best overall quality, math, tables | Cloud API, ~$0.001/page | +| **DeepSeek OCR** | `pip install "openaireview[deepseek]"` + local backend | Privacy-sensitive docs | Local model via Ollama/vLLM | +| **Marker** | `uv tool install marker-pdf --with psutil` | Math-heavy PDFs (offline) | Slow without GPU | +| **pymupdf4llm** | (included) | Fallback, always available | No math symbol support | + The engine is auto-detected: if `MISTRAL_API_KEY` is set, Mistral OCR is tried first; then DeepSeek (if installed); then Marker (if on PATH); finally pymupdf4llm. You can force a specific engine with `--ocr`: @@ -96,34 +100,40 @@ openaireview serve Review an academic paper for technical and logical issues. Accepts a local file path or an arXiv URL. -| Option | Default | Description | -|---|---|---| -| `--method` | `progressive` | Review method: `zero_shot`, `local`, `progressive`, `progressive_full` | -| `--model` | `anthropic/claude-opus-4-6` | Model to use | -| `--provider` | (auto) | LLM provider: `openrouter`, `openai`, `anthropic`, `gemini`, `mistral` | -| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | -| `--max-pages` | (all) | Only process first N pages of a PDF (saves OCR cost) | -| `--max-tokens` | (all) | Truncate input text to first N tokens before review | -| `--output-dir` | `./review_results` | Directory for output JSON files | -| `--name` | (from filename) | Paper slug name | + +| Option | Default | Description | +| -------------- | --------------------------- | ---------------------------------------------------------------------- | +| `--method` | `progressive` | Review method: `zero_shot`, `local`, `progressive`, `progressive_full` | +| `--model` | `anthropic/claude-opus-4-6` | Model to use | +| `--provider` | (auto) | LLM provider: `openrouter`, `openai`, `anthropic`, `gemini`, `mistral` | +| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | +| `--max-pages` | (all) | Only process first N pages of a PDF (saves OCR cost) | +| `--max-tokens` | (all) | Truncate input text to first N tokens before review | +| `--output-dir` | `./review_results` | Directory for output JSON files | +| `--name` | (from filename) | Paper slug name | + ### `openaireview extract ` Run OCR extraction only and save as markdown with metadata frontmatter. Useful for a two-stage workflow: extract first, then review the markdown. -| Option | Default | Description | -|---|---|---| -| `-o`, `--output` | `.md` | Output markdown path | -| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | + +| Option | Default | Description | +| ---------------- | ----------- | ---------------------------------------------------------- | +| `-o`, `--output` | `.md` | Output markdown path | +| `--ocr` | (auto) | PDF OCR engine: `mistral`, `deepseek`, `marker`, `pymupdf` | + ### `openaireview serve` Start a local visualization server to browse review results. -| Option | Default | Description | -|---|---|---| + +| Option | Default | Description | +| --------------- | ------------------ | -------------------------------------- | | `--results-dir` | `./review_results` | Directory containing result JSON files | -| `--port` | `8080` | Server port | +| `--port` | `8080` | Server port | + ## Supported Input Formats @@ -135,15 +145,17 @@ Start a local visualization server to browse review results. ## Environment Variables -| Variable | Default | Description | -|---|---|---| -| `OPENROUTER_API_KEY` | | OpenRouter API key (supports all models) | -| `OPENAI_API_KEY` | | OpenAI native API key | -| `ANTHROPIC_API_KEY` | | Anthropic native API key | -| `GEMINI_API_KEY` | | Google Gemini native API key | -| `MISTRAL_API_KEY` | | Mistral API key (also used for Mistral OCR) | -| `MODEL` | `anthropic/claude-opus-4-6` | Default model | -| `REVIEW_PROVIDER` | (auto) | Force a specific LLM provider | + +| Variable | Default | Description | +| -------------------- | --------------------------- | ------------------------------------------- | +| `OPENROUTER_API_KEY` | | OpenRouter API key (supports all models) | +| `OPENAI_API_KEY` | | OpenAI native API key | +| `ANTHROPIC_API_KEY` | | Anthropic native API key | +| `GEMINI_API_KEY` | | Google Gemini native API key | +| `MISTRAL_API_KEY` | | Mistral API key (also used for Mistral OCR) | +| `MODEL` | `anthropic/claude-opus-4-6` | Default model | +| `REVIEW_PROVIDER` | (auto) | Force a specific LLM provider | + Set one API key. The provider is auto-detected from whichever key is set (priority: OpenRouter > OpenAI > Anthropic > Gemini > Mistral). See `.env.example` for a template. @@ -151,12 +163,14 @@ Set one API key. The provider is auto-detected from whichever key is set (priori All models available on [OpenRouter](https://openrouter.ai) are supported — use any model ID via `--model`. The following models have built-in pricing for accurate cost tracking in the visualization: -| Model | Input ($/1M tokens) | Output ($/1M tokens) | -|---|---|---| -| `anthropic/claude-opus-4-6` | $5.00 | $25.00 | -| `anthropic/claude-opus-4-5` | $5.00 | $25.00 | -| `openai/gpt-5.2-pro` | $21.00 | $168.00 | -| `google/gemini-3.1-pro-preview` | $2.00 | $12.00 | + +| Model | Input ($/1M tokens) | Output ($/1M tokens) | +| ------------------------------- | ------------------- | -------------------- | +| `anthropic/claude-opus-4-6` | $5.00 | $25.00 | +| `anthropic/claude-opus-4-5` | $5.00 | $25.00 | +| `openai/gpt-5.2-pro` | $21.00 | $168.00 | +| `google/gemini-3.1-pro-preview` | $2.00 | $12.00 | + For models not listed above, a default rate of $5.00/$25.00 per 1M tokens is used. @@ -214,31 +228,31 @@ export OPENROUTER_API_KEY=... Run scripts from inside each benchmark's directory unless noted. -### Outcomes study (`benchmarks/conference_study/`) +### Quality-proxy study (`benchmarks/conference_study/`) -Compares OpenAIReview output on accepted vs. rejected conference submissions. Papers are sampled via the 4-pair SNOR signal matrix (top-cited vs. never-published, awarded vs. rejected, top vs. bottom scores, and a composed pair). +Tests whether AI review systems comment more on weaker papers than on stronger ones. Papers are split into high- and low-quality groups by four noisy quality proxies built from citations, venue awards, review scores, and a composite of the three. `configs/baseline.yaml` (our system) and `coarse.yaml` (coarse, an external review system) are the committed example configs. ```bash cd benchmarks/conference_study -# 1. Build manifests (manifests/v1/{pair_1..4,combined}.json) -python select_papers.py --venues iclr neurips --years 2021 2022 +# 1. Build the canonical paper manifest from SNOR (deterministic; downloads ~448 MB once) +python select_papers.py # writes manifests/canonical/ -# 2. Download PDFs flat under papers/scaleup/, write pages back into the manifest -python download_papers.py --source snor +# 2. Download PDFs into papers/, writing pages back into the manifest +python download_papers.py --source snor --manifest manifests/canonical/full.json -# 3. Optional cost preview (drops PDF parsing; estimate = pages × tokens_per_page × multipliers) -python estimate_cost.py --config configs/scaleup_progressive.yaml +# 3. Optional cost preview (estimate = pages × tokens_per_page × multipliers) +python estimate_cost.py --config configs/baseline.yaml -# 4. Run OpenAIReview and/or competitor systems on the same paper × model grid -python run_study.py --config configs/scaleup_progressive.yaml -python run_competitors.py --config configs/coarse_v2.yaml +# 4. Run our system and/or external systems on the same paper × model grid +python run_study.py --config configs/baseline.yaml +python run_competitors.py --config configs/coarse.yaml -# 5. Aggregate -python analyses/report_scaleup.py results/scaleup_progressive +# 5. Aggregate into the tables reported in the paper +python analyses/generate_report.py --config configs/baseline.yaml ``` -`run_study.py` and `run_competitors.py` are idempotent — rerunning skips paper × model combos already complete. Per-paper locks let multiple models share the same result JSON. See `benchmarks/conference_study/README.md` for the config schema, concurrency model, and result format. +The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontier subset the paper mainly reports on has no regeneration script and is not shipped; contact the authors for its manifest to reproduce those results. `run_study.py` and `run_competitors.py` are idempotent, so rerunning skips paper × model combos already complete. See `benchmarks/conference_study/README.md` for more details, include how to run `coarse`. ### Perturbation benchmark (`benchmarks/perturbation/`) @@ -273,4 +287,4 @@ The config picks the review system per run via `system: openaireview | coarse | ## License -[MIT](LICENSE) +[MIT](LICENSE) \ No newline at end of file diff --git a/benchmarks/README.md b/benchmarks/README.md deleted file mode 100644 index dfebe75..0000000 --- a/benchmarks/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# Benchmarks - -Reproducibility code for the two studies in the paper: - -- **Outcomes study** ([`conference_study/`](conference_study/)) — Does the proposed reviewer produce more (or qualitatively different) comments on rejected vs. accepted conference submissions? Papers are sampled via a 4-pair signal matrix (top-cited vs. never-published, awarded vs. rejected, top vs. bottom review scores, and a composed pair). -- **Perturbation benchmark** ([`perturbation/`](perturbation/)) — Can the proposed reviewer detect *seeded* errors injected into clean papers? Errors span four categories: surface math edits, false claims, faulty reasoning, and experimental-design flaws. Recall is reported per (model, method) cell. - -Both share a common setup. Throughout this document, the **system under evaluation** is the proposed reviewer (referred to as a "system" or "method" below); **competitor systems** are external review tools we compare against. - -## Setup - -```bash -uv pip install -e ".[benchmarks]" -export OPENROUTER_API_KEY=... # all model calls route through OpenRouter -``` - -Run scripts from inside each benchmark's directory. - -## Outcomes study - -Configs that produced the main-text tables: `conference_study/configs/scaleup_progressive.yaml` (proposed system) and `conference_study/configs/coarse_v2.yaml` (competitor). End-to-end: - -```bash -cd conference_study - -# 1. Build manifests (manifests/v1/{pair_1..4,combined}.json) -python select_papers.py --venues iclr neurips --years 2021 2022 - -# 2. Download PDFs flat under papers/scaleup/, write pages back into the manifest -python download_papers.py --source snor - -# 3. Optional cost preview (estimate = pages × tokens_per_page × method-specific multipliers) -python estimate_cost.py --config configs/scaleup_progressive.yaml - -# 4. Run the proposed system and/or competitor systems on the same paper × model grid -python run_study.py --config configs/scaleup_progressive.yaml -python run_competitors.py --config configs/coarse_v2.yaml - -# 5. Aggregate into the tables reported in the paper -python analyses/report_scaleup.py results/scaleup_progressive -``` - -Both runners are idempotent — rerunning skips (paper × model) combos already complete. Per-paper locks let multiple models share the same result JSON. See [`conference_study/README.md`](conference_study/README.md) for the config schema, concurrency model, and result format. - -## Perturbation benchmark - -Pipeline: `extract → generate → validate → verify → inject → review → score`. The first five stages produce a *corrupted paper* and a ground-truth manifest of injected errors; the last two stages run the reviewer and score its output. `run_benchmark.py` drives all stages from a single YAML. - -```bash -cd perturbation - -# One-shot: prepare papers, run reviews, score against the perturbation manifest -python run_benchmark.py configs/default.yaml - -# Or run a subset of stages (useful when iterating on scoring or rerunning a single model) -python run_benchmark.py configs/default.yaml --stages prepare,review -python run_benchmark.py configs/default.yaml --stages score - -# Multi-config sweep with parallel workers reused across configs -python run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 - -# Aggregate the recall-by-{model,method,error-type,domain} tables in the paper -python generate_report.py results/ -``` - -The `system:` field in each config selects the reviewer under test. Scoring uses a two-stage filter: (1) a fuzzy substring match on the perturbed text against each review comment's quote (after whitespace + math-delimiter normalization, with a 0.75 coverage threshold), then (2) an LLM judge rating (≥3/5) on whether the comment's explanation identifies the same error described in the perturbation's `why_wrong` field. See [`perturbation/README.md`](perturbation/README.md) for the error-type taxonomy, results layout, and known limitations. - -## Repository map - -``` -benchmarks/ -├── conference_study/ # outcomes study (§ above) -│ ├── select_papers.py # build the 4-pair signal-matrix manifest -│ ├── download_papers.py # fetch PDFs from OpenReview -│ ├── run_study.py # batch runner, proposed system -│ ├── run_competitors.py # batch runner, external systems -│ ├── analyses/ # report-generation scripts -│ └── configs/ # per-experiment YAMLs -└── perturbation/ # perturbation benchmark (§ above) - ├── run_benchmark.py # single-entry pipeline driver - ├── extract.py / generate.py / validate.py / verify.py / inject.py - ├── score.py / generate_report.py - ├── systems/ # adapters for the proposed system + competitors - └── configs/ # per-experiment YAMLs -``` - -The reviewer code itself is at the repository root (`src/`), exposed as a Python package and a CLI; both benchmarks invoke it through their adapter layers rather than calling its internals directly. diff --git a/benchmarks/conference_study/.gitignore b/benchmarks/conference_study/.gitignore index 215905b..58d4641 100644 --- a/benchmarks/conference_study/.gitignore +++ b/benchmarks/conference_study/.gitignore @@ -8,5 +8,6 @@ papers/ # Local configs — ignore all, un-ignore the canonical ones. configs/*.yaml !configs/baseline.yaml +!configs/frontier.yaml !configs/coarse.yaml !configs/reviewer3.yaml diff --git a/benchmarks/conference_study/README.md b/benchmarks/conference_study/README.md index c696800..b2b5ce0 100644 --- a/benchmarks/conference_study/README.md +++ b/benchmarks/conference_study/README.md @@ -1,177 +1,167 @@ -# Conference Study: Accepted vs Rejected +# Quality-Proxy Study -**Research question:** Does OpenAIReview's progressive method produce more (or -different) comments on rejected papers than on accepted ones? +Do AI review systems comment more on weaker papers than on stronger ones? +This study tests whether comment volume tracks paper quality on real +ICLR/NeurIPS papers. Paper quality has no gold-standard label, so we split +papers into high- and low-quality groups using four noisy **quality proxies** +built from citation, award, and review-score signals, and check whether each +system produces more comments on the low-quality group. -**Venue:** ICLR 2024. The only major ML venue where rejected submissions are -publicly available on OpenReview. NeurIPS, ICML, CVPR etc. hide rejected -papers, so this study can't easily port to them. +The paper has the full method, metric, and results +([arXiv:2606.19749](https://arxiv.org/abs/2606.19749), Section "AI Review +Systems Correlate with Human Quality Signals"). This README covers how to +reproduce the runs. -## Directory layout - -``` -configs/ # YAML run configs (one per experiment) -reports/ # Markdown write-ups (one per experiment) -results/ # Output JSONs + run_log.jsonl (gitignored) -papers/ # Downloaded PDFs (gitignored; regenerate via download_papers.py) - accepted/ # 5 ICLR 2024 Outstanding Paper Award winners - rejected/ # 5 substantive rejections (ratings 2.5–3.5, ≥3 reviewers) -competitors/ # Adapters for external review systems (coarse, etc.) -download_papers.py # Fetches PDFs + writes manifest.json -estimate_cost.py # Pre-run USD cost estimate (progressive method) -run_study.py # Batch runner for OpenAIReview progressive (multi-model, multi-paper) -run_competitors.py # Batch runner for competitor systems (same paper/model grid) -generate_report.py # Auto-detects systems and emits per-system summary tables -manifest.json # Curated paper list (forum IDs, titles, groups) -``` +## The four quality proxies -One experiment = `configs/.yaml` + `reports/.md` + -`results//`. The triad shares a name. +Each proxy splits papers into a high-quality and a low-quality group of 30, +drawn from ICLR/NeurIPS 2021-2022 papers with at least three reviews and a +non-null average score: -## Workflow +- **Community-level**: top 30 by citations-per-year vs 30 rejected papers never published elsewhere. +- **Conference-level**: 30 papers highlighted by the venue (Outstanding, Best, Oral, Spotlight) vs 30 rejected papers. +- **Reviewer-level**: top 30 by mean review score vs bottom 30. +- **Composite**: top 30 awarded papers by combined citation-and-score rank vs bottom 30 rejected, never-published papers by review score. -### 1. Download papers (one-time) - -PDFs are gitignored. Download them before the first run: - -```bash -python download_papers.py -``` +That gives 60 papers per proxy and 240 in total (197 unique, since some papers +satisfy more than one criterion). Frontier models too expensive to run on the +full set use a **frontier subset** of 74 unique papers, which the paper mainly +reports on. -If `manifest.json` exists, the script downloads the exact papers listed -there, skipping any already on disk. The manifest is never modified. +Citations, awards, and review scores are noisy proxies of quality. We pick the +top and bottom groups as a tractable approximation, not as ground-truth quality. -**Adding more papers** to an existing manifest: +## Reproducing the paper set -```bash -python download_papers.py --add -n 5 # 5 more per group (papercopilot) -python download_papers.py --add -n 3 --group rejected # 3 more rejected only -``` +Selection draws from SNOR ([Zenodo 15866613](https://zenodo.org/records/15866613)), +which links OpenReview submissions to Semantic Scholar citation counts, +decisions, and reviewer scores. -**Bootstrapping a new study** (no manifest yet): +**Full set (197 papers)** regenerates from code: ```bash -python download_papers.py -n 10 # papercopilot, ICLR 2024 -python download_papers.py --source hf --venue ICLR --year 2023 -n 10 # HuggingFace -python download_papers.py --source hf --venue NeurIPS --year 2022 -n 10 +cd conference_study +python select_papers.py # downloads SNOR (~448 MB, cached), writes manifests/canonical/ ``` -Two data sources via `--source`: - -- **papercopilot** (default): ICLR only, uses explicit accept/reject decisions. -- **hf**: [AlgorithmicResearchGroup/openreview-papers-with-reviews](https://huggingface.co/datasets/AlgorithmicResearchGroup/openreview-papers-with-reviews) - on HuggingFace. Multi-venue (ICLR, NeurIPS, CoRL, UAI, MIDL), selects - by review score thresholds (`--accepted-threshold`, `--rejected-threshold`) - since the dataset has no explicit decisions. - -### 2. Estimate cost before a run - -```bash -python estimate_cost.py -``` +Selection is deterministic (fixed seed, forum-id tiebreak), so this reproduces +the exact 197-paper set used in the paper. -Uses observed progressive-method token multipliers (~6× input for the chain -of running-summary + window replay, ~0.15× output). Sanity-check before -burning API credits. +**Frontier subset (74 papers)** is a fixed subsample of the full set with no +regeneration script, so it is not shipped here. The pipeline runs end to end on +the full set with the repo code; to reproduce the paper's frontier-subset +numbers, contact the authors for `manifests/canonical/frontier.json`. -### 3. Run an experiment +## Run flow ```bash -export OPENROUTER_API_KEY=... -python run_study.py --config configs/baseline.yaml -``` +cd conference_study +export OPENROUTER_API_KEY=... # all model calls route through OpenRouter -Writes results to `results//.json` and a run log to -`results//run_log.jsonl`. Idempotent — rerunning skips paper/model -combos already complete. +# 1. Build the full paper manifest. +python select_papers.py -**Useful flags:** +# 2. Download the PDFs listed in a manifest. Add --limit 5 for a cheap end-to-end smoke test first. +python download_papers.py --source snor --manifest manifests/canonical/full.json -- `--dry-run` — print the commands without calling the API. -- `--paper ` / `--model ` — restrict to one paper or model. -- `--force` — re-run even if already complete. -- `--max-pages` / `--max-tokens` / `--timeout-sec` / `--max-per-model` — ad-hoc -overrides of any YAML value. +# 3. Preview the API cost before the full run. estimate_cost.py prints a per-model +# breakdown; the full efficient set costs ~$25-50. +python estimate_cost.py --config configs/baseline.yaml -### 4. Run a competitor system (optional) +# 4. Run the systems on the same paper x model grid. +python run_study.py --config configs/baseline.yaml # our system, efficient models, full set +python run_competitors.py --config configs/coarse.yaml # coarse, in its own venv (see "Running coarse") -Competitors are external review systems wrapped via adapters in -`competitors/`. They run on the same paper × model grid as `run_study.py` -and their outputs merge into the same result-JSON schema, so the report -tooling handles them alongside OpenAIReview's progressive. - -```bash -python run_competitors.py --config configs/coarse.yaml +# 5. Build the paper's tables (pairwise accuracy + 95% CIs) from the results. +# Pass several configs to compare systems in one set of tables. +python analyses/generate_report.py --config configs/baseline.yaml configs/coarse.yaml ``` -Each competitor has its own config (e.g. `configs/coarse.yaml`) with a -`competitor:` field naming the adapter. Adapters live in -`competitors/_adapter.py` and are registered in -`competitors/registry.py`. See that file's docstring for how to add a new -one. +Start with the `--limit 5` smoke run in step 2 plus one model to confirm the +pipeline end to end for a few cents before launching the full set. Both runners +are idempotent: rerunning skips (paper, model) combos already complete. -Competitor outputs get their own `results//` directory, -parallel to `results//`. To report on both, point -`generate_report.py` at the combined results directory or at each -separately. +To reproduce the paper's frontier-subset numbers (all six models, including the +two frontier ones), obtain `manifests/canonical/frontier.json` from the authors, +then run steps 4 and 5 with `configs/frontier.yaml` in place of `baseline.yaml`. -### 5. Generate report tables +## Running coarse + +`coarse` (PyPI `coarse-ink`) has a large, version-sensitive dependency set, so +it runs in its own virtual environment and the adapter subprocess-calls into it. +Set it up once: ```bash -python generate_report.py --config configs/baseline.yaml -python generate_report.py --config configs/baseline.yaml --papers # include papers table (parses PDFs, slow) +uv venv /path/to/coarse-venv +uv pip install --python /path/to/coarse-venv coarse-ink +export COARSE_VENV_PYTHON=/path/to/coarse-venv/bin/python ``` -Auto-detects every system present in the result JSONs (e.g. `progressive`, -`coarse`) by scanning method prefixes, and emits a per-system block of -tables (overall, per-model, consolidation if applicable, cost) plus a -runtime table spanning all systems. Pipe to a file or paste into -`reports/.md`. +`run_competitors.py --config configs/coarse.yaml` finds the venv through +`COARSE_VENV_PYTHON` (or a `venv_python:` field in the config). coarse runs on +the same paper x model grid, and its output merges into the same result schema +as the other systems. -### 6. Add a new experiment +## Metric -```bash -cp configs/baseline.yaml configs/my_experiment.yaml -# edit name, caps, parallelism as needed -python run_study.py --config configs/my_experiment.yaml -python generate_report.py --config configs/my_experiment.yaml -# write up findings in reports/my_experiment.md -``` +For each (system, model, proxy) we compare mean comments on the high- and +low-quality groups and report **pairwise accuracy**: the fraction of (low, +high) paper pairs where the low-quality paper received more comments (ties +count 0.5). A value of 0.5 means no separation, and above 0.5 means the system +tracks the proxy direction. Confidence intervals come from a cluster bootstrap +over papers (`analyses/ci_auc.py`); see the paper's appendix for the +construction. ## Config schema ```yaml -name: baseline # Results -> results// -max_pages: 20 # Passed to openaireview CLI -max_tokens: 20000 # Passed to openaireview CLI -timeout_sec: 3600 # Per-subprocess timeout -max_per_model: 2 # Concurrent runs per model +name: baseline # results -> results// +manifest: manifests/canonical/full.json # paper set to run on +models: # backbone models (falls back to the manifest's list if omitted) + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b +max_pages: 20 # passed to the openaireview CLI +max_tokens: 20000 # passed to the openaireview CLI +timeout_sec: 3600 # per-subprocess timeout +max_per_model: 2 # concurrent runs per model ``` -Precedence: **CLI flag > YAML value > built-in default**. CLI flag names -mirror YAML keys (hyphens for underscores). +## Directory layout -## Concurrency model +``` +configs/ # run configs (baseline = full set, frontier = 74-subset, coarse = external system) +manifests/ # paper set (canonical/full.json regenerates via select_papers.py) +papers/ # downloaded PDFs (gitignored; regenerate via download_papers.py) +results/ # output JSONs + run_log.jsonl (gitignored) +competitors/ # adapters for external review systems +analyses/ # report + AUC + confidence-interval scripts +select_papers.py # build the canonical manifest from SNOR +download_papers.py # fetch PDFs listed in a manifest +run_study.py # batch runner for our system +run_competitors.py # batch runner for external review systems +``` -Each model gets its own queue + `max_per_model` worker threads — keeps -OpenRouter per-model rate limits isolated. Per-paper locks prevent two -models from clobbering the same result JSON during the CLI's -read-modify-write merge. Total in-flight = `max_per_model × num_models` -(default 6 for 3 models). +## Output structure -## Results format +Each run config writes to its own results directory: -Each `results//.json` contains a `methods` dict keyed by -`__`. OpenAIReview's progressive writes two keys: +``` +results// + .json # one file per paper, with every model and system merged in + run_log.jsonl # one line per (paper, model) run: timing, exit code, output tails +``` -- `progressive__` — comments after the consolidation pass. -- `progressive_original__` — raw comments before consolidation. +Each `.json` holds the paper identity (`slug`, `title`), the parsed +`paragraphs` the systems reviewed, and a `methods` dict keyed by +`__`: -Both are written in a single CLI invocation. `run_study.py` considers a -(paper, model) combo "done" only when both keys exist. +``` +methods: + progressive__ # OpenAIReview comments after consolidation + progressive_original__ # OpenAIReview comments before consolidation + coarse__ # one key per external system run +``` -Competitor systems typically write a single post-editorial key -`__` (no raw/consolidated pair). The report -generator scans the method keys to decide which tables apply to which -system — systems without a `_original` partner skip the consolidation -table. \ No newline at end of file +Each method entry has `label`, `model`, `overall_feedback`, `cost_usd`, +`prompt_tokens`, `completion_tokens`, and a `comments` list. Each comment has a +`title`, `quote`, `explanation`, `paragraph_index`, and `severity`. \ No newline at end of file diff --git a/benchmarks/conference_study/analyses/compute_auc.py b/benchmarks/conference_study/analyses/compute_auc.py index 3ff04a6..11293ee 100644 --- a/benchmarks/conference_study/analyses/compute_auc.py +++ b/benchmarks/conference_study/analyses/compute_auc.py @@ -1,5 +1,5 @@ #!/usr/bin/env python -"""Mann-Whitney AUC for the conference outcomes study. +"""Mann-Whitney AUC for the quality-proxy study. For each (method, model) found in the supplied result directories, computes: @@ -190,7 +190,7 @@ def main() -> int: """CLI: load the manifest and result dirs, compute each (method, model) cell's point summary, and print them as a markdown table.""" p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--manifest", required=True, type=Path, help="Path to manifest JSON (e.g., manifests/v2/combined.json).") + p.add_argument("--manifest", required=True, type=Path, help="Path to manifest JSON (e.g., manifests/canonical/full.json).") p.add_argument("--dir", nargs="+", required=True, dest="dirs", help="One or more result subdirectories under results/ (e.g., scaleup_v2_progressive).") p.add_argument("--method", default=None, help="Filter to one method (e.g., progressive_original, zero_shot, coarse).") diff --git a/benchmarks/conference_study/analyses/generate_report.py b/benchmarks/conference_study/analyses/generate_report.py index 99e5b51..ce2bf6c 100644 --- a/benchmarks/conference_study/analyses/generate_report.py +++ b/benchmarks/conference_study/analyses/generate_report.py @@ -1,536 +1,108 @@ #!/usr/bin/env python -"""Compute summary tables from conference study results. - -Reads the result JSONs and run log to produce the markdown tables found in -reports/baseline.md. Can optionally parse PDFs to compute per-paper page -counts and effective token counts (requires papers/ on disk). +"""Produce the paper's quality-proxy tables from a run's results. + +Reads the result JSONs for a run (selected by --config or --results-dir) and the +manifest's high/low group memberships, then prints the pairwise-accuracy +(Mann-Whitney AUC) tables with 95% cluster-bootstrap confidence intervals: the +per-model comment-volume table, the per-quality-proxy table, and the +per-severity-tier table. + +The (method, model) cells are discovered from whatever is present in the +results, so no separate table config is needed: run the systems (step 4), then +point this at their results directories. Pass several to compare systems in one +table. By default it shows the manifest's roster models and openaireview's +pre-consolidation progressive output (the variant the paper reports). Use +--all-models and --consolidated to change that. The AUC and bootstrap math lives +in compute_auc.py and ci_auc.py, and this script is the reporting entry point +over them. Usage: - python generate_report.py # legacy results/ - python generate_report.py --config configs/baseline.yaml # results// - python generate_report.py --results-dir results/baseline # explicit + python analyses/generate_report.py --config configs/baseline.yaml + # merge systems into one set of tables + python analyses/generate_report.py --config configs/baseline.yaml configs/coarse.yaml + python analyses/generate_report.py --results-dir results/baseline results/coarse \ + --manifest manifests/canonical/full.json """ from __future__ import annotations import argparse import json import sys -from collections import defaultdict from pathlib import Path import yaml -HERE = Path(__file__).resolve().parent -REPO = HERE.parent.parent - -# Models in display order for the baseline three. Any model not in this list -# still appears (sorted alphabetically after these). -MODEL_ORDER = ["gemini-3-flash-preview", "glm-4.6", "qwen3-235b-a22b-2507"] - - -def ordered_models(models: set[str]) -> list[str]: - """MODEL_ORDER first, then any others sorted.""" - head = [m for m in MODEL_ORDER if m in models] - tail = sorted(m for m in models if m not in MODEL_ORDER) - return head + tail - - -# --------------------------------------------------------------------------- -# System detection -# --------------------------------------------------------------------------- -# -# A "system" is a review producer (e.g. `progressive`, `coarse`). Each system -# writes methods of the form `__` into result JSONs. Some -# systems (openaireview progressive) also emit a raw-vs-consolidated pair -# (`progressive_original__` alongside `progressive__`); others -# (competitors) emit only a single prefix. Detection: scan the actual method -# keys and pair up any `*_original` with its base. - -def detect_systems(results: dict[str, dict]) -> list[dict]: - """Return [{name, cons_prefix, raw_prefix | None}, ...] detected from JSONs.""" - prefixes: set[str] = set() - for res in results.values(): - for key in res.get("methods", {}): - if "__" in key: - prefixes.add(key.split("__", 1)[0]) - - systems = [] - for p in sorted(prefixes): - if p.endswith("_original"): - continue # shown via its consolidated partner's shrink column - raw = f"{p}_original" - systems.append({ - "name": p, - "cons_prefix": p, - "raw_prefix": raw if raw in prefixes else None, - }) - return systems - - -# --------------------------------------------------------------------------- -# Config loading (same pattern as run_study.py / estimate_cost.py) -# --------------------------------------------------------------------------- - -def load_config(path: str) -> dict: - if not path: - return {} - cfg_path = Path(path) - if not cfg_path.is_absolute(): - cfg_path = cfg_path if cfg_path.exists() else HERE / path - if not cfg_path.exists(): - sys.exit(f"config file not found: {path}") - with cfg_path.open() as f: - return yaml.safe_load(f) or {} - - -# --------------------------------------------------------------------------- -# Data loading -# --------------------------------------------------------------------------- - -def load_results(results_dir: Path, manifest: dict) -> dict[str, dict]: - """Load result JSONs for all papers in the manifest.""" - results = {} - for p in manifest["papers"]: - slug = p["slug"] - path = results_dir / f"{slug}.json" - if not path.exists(): - print(f" warning: missing {path}", file=sys.stderr) - continue - results[slug] = json.loads(path.read_text()) - return results - - -def load_run_log(results_dir: Path) -> list[dict]: - """Load run_log.jsonl entries.""" - log_path = results_dir / "run_log.jsonl" - if not log_path.exists(): - return [] - entries = [] - for line in log_path.read_text().splitlines(): - if line.strip(): - entries.append(json.loads(line)) - return entries - - -def model_short(model: str) -> str: - return model.split("/")[-1] if "/" in model else model - - -def paper_group(slug: str, manifest: dict) -> str: - for p in manifest["papers"]: - if p["slug"] == slug: - return p["group"] - return "unknown" - - -# --------------------------------------------------------------------------- -# Papers table (requires PDFs on disk) -# --------------------------------------------------------------------------- - -def compute_papers_table(manifest: dict, max_pages: int, max_tokens: int) -> list[dict]: - """Parse PDFs to get page counts and effective token counts.""" - sys.path.insert(0, str(REPO / "src")) - from reviewer.parsers import parse_document - from reviewer.utils import count_tokens - - rows = [] - for p in manifest["papers"]: - slug = p["slug"] - pdf = HERE / "papers" / p["group"] / f"{slug}.pdf" - if not pdf.exists(): - rows.append({"slug": slug, "group": p["group"], - "title": p["title"], "pages": None, "eff_tokens": None}) - continue - try: - import fitz - doc = fitz.open(pdf) - pages = len(doc) - doc.close() - except Exception: - pages = None - - try: - _title, content, _ocr = parse_document(pdf, max_pages=max_pages) - eff_tokens = min(count_tokens(content), max_tokens) - except Exception: - eff_tokens = None - - rows.append({"slug": slug, "group": p["group"], - "title": p["title"], "pages": pages, "eff_tokens": eff_tokens}) - return rows - - -# --------------------------------------------------------------------------- -# Comment counting helpers -# --------------------------------------------------------------------------- - -def count_comments(result: dict, method_prefix: str) -> dict[str, int]: - """Count comments per model for a given method prefix. - - Returns {model_short: total_count}. - """ - counts: dict[str, int] = {} - methods = result.get("methods", {}) - for key, data in methods.items(): - if not key.startswith(method_prefix + "__"): - continue - model = key[len(method_prefix) + 2:] - counts[model] = len(data.get("comments", [])) - return counts - - -def get_cost(result: dict, method_prefix: str) -> dict[str, float]: - """Return {model_short: cost_usd} for the given prefix.""" - costs: dict[str, float] = {} - methods = result.get("methods", {}) - for key, data in methods.items(): - if not key.startswith(method_prefix + "__"): - continue - model = key[len(method_prefix) + 2:] - costs[model] = data.get("cost_usd") or 0.0 - return costs - - -# --------------------------------------------------------------------------- -# Table builders -# --------------------------------------------------------------------------- - -def build_overall_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Overall: accepted vs rejected, for one system.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - groups: dict[str, dict] = { - "accepted": {"n": 0, "raw": 0, "consolidated": 0}, - "rejected": {"n": 0, "raw": 0, "consolidated": 0}, - } - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) if raw_pref else cons_counts - for model, cons in cons_counts.items(): - groups[group]["n"] += 1 - groups[group]["consolidated"] += cons - groups[group]["raw"] += raw_counts.get(model, cons) - - rows = [] - for g in ["accepted", "rejected"]: - d = groups[g] - n = d["n"] - if n == 0: - continue - raw_avg = d["raw"] / n - cons_avg = d["consolidated"] / n - shrink = 1 - cons_avg / raw_avg if raw_avg > 0 else 0 - rows.append({"group": g, "n": n, - "raw_total": d["raw"], "raw_avg": raw_avg, - "cons_total": d["consolidated"], "cons_avg": cons_avg, - "shrink": shrink}) - return rows +import ci_auc +from compute_auc import load_counts, load_manifest +HERE = Path(__file__).resolve().parent +STUDY = HERE.parent # benchmarks/conference_study -def build_per_model_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Per-model breakdown for one system.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - cells: dict[tuple[str, str], dict] = defaultdict( - lambda: {"n": 0, "raw": 0, "consolidated": 0} - ) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) if raw_pref else cons_counts - for model, cons in cons_counts.items(): - seen_models.add(model) - cells[(group, model)]["n"] += 1 - cells[(group, model)]["consolidated"] += cons - cells[(group, model)]["raw"] += raw_counts.get(model, cons) - - rows = [] - for group in ["accepted", "rejected"]: - for model in ordered_models(seen_models): - d = cells.get((group, model)) - if not d or d["n"] == 0: - continue - raw_avg = d["raw"] / d["n"] - cons_avg = d["consolidated"] / d["n"] - shrink = 1 - cons_avg / raw_avg if raw_avg > 0 else 0 - rows.append({"group": group, "model": model, "n": d["n"], - "raw_total": d["raw"], "raw_avg": raw_avg, - "cons_total": d["consolidated"], "cons_avg": cons_avg, - "shrink": shrink}) - return rows - - -def build_consolidation_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Consolidation shrink per model, split by group. Only meaningful when - the system exposes a raw_prefix.""" - cons_pref = system["cons_prefix"] - raw_pref = system["raw_prefix"] - if not raw_pref: - return [] - cells: dict[tuple[str, str], dict] = defaultdict( - lambda: {"raw": 0, "consolidated": 0} - ) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - cons_counts = count_comments(res, cons_pref) - raw_counts = count_comments(res, raw_pref) - for model in raw_counts: - seen_models.add(model) - cells[(group, model)]["raw"] += raw_counts[model] - cells[(group, model)]["consolidated"] += cons_counts.get(model, 0) - - rows = [] - for model in ordered_models(seen_models): - acc = cells.get(("accepted", model), {"raw": 0, "consolidated": 0}) - rej = cells.get(("rejected", model), {"raw": 0, "consolidated": 0}) - acc_shrink = 1 - acc["consolidated"] / acc["raw"] if acc["raw"] > 0 else 0 - rej_shrink = 1 - rej["consolidated"] / rej["raw"] if rej["raw"] > 0 else 0 - rows.append({"model": model, "acc_shrink": acc_shrink, "rej_shrink": rej_shrink}) - return rows - - -def build_cost_table( - manifest: dict, results: dict[str, dict], system: dict -) -> list[dict]: - """Cost per model, split by group, for one system.""" - cells: dict[tuple[str, str], float] = defaultdict(float) - seen_models: set[str] = set() - for slug, res in results.items(): - group = paper_group(slug, manifest) - costs = get_cost(res, system["cons_prefix"]) - for model, cost in costs.items(): - seen_models.add(model) - cells[(group, model)] += cost - - rows = [] - for model in ordered_models(seen_models): - acc = cells.get(("accepted", model), 0.0) - rej = cells.get(("rejected", model), 0.0) - rows.append({"model": model, "accepted": acc, "rejected": rej, - "total": acc + rej}) - return rows - - -def build_runtime_table(run_log: list[dict], manifest: dict) -> list[dict]: - """Runtime per model from run_log.jsonl.""" - manifest_slugs = {p["slug"] for p in manifest["papers"]} - # {model_short: [durations]} - durations: dict[str, list[float]] = defaultdict(list) - for entry in run_log: - if not entry.get("ok") or entry["slug"] not in manifest_slugs: - continue - model = model_short(entry["model"]) - durations[model].append(entry["duration_sec"]) - - rows = [] - for model in ordered_models(set(durations)): - durs = durations.get(model, []) - if not durs: - continue - avg_min = (sum(durs) / len(durs)) / 60 - min_min = min(durs) / 60 - max_min = max(durs) / 60 - rows.append({"model": model, "avg_min": avg_min, - "min_min": min_min, "max_min": max_min}) - return rows - - -# --------------------------------------------------------------------------- -# Markdown formatting -# --------------------------------------------------------------------------- - -def fmt_papers(rows: list[dict]) -> str: - lines = [] - lines.append("| Slug | Group | Title | Pages | Eff. tokens |") - lines.append("| --- | --- | --- | --- | --- |") - for r in rows: - pages = str(r["pages"]) if r["pages"] is not None else "N/A" - tokens = f"{r['eff_tokens']:,}" if r["eff_tokens"] is not None else "N/A" - lines.append(f"| {r['slug']} | {r['group']} | {r['title']} | {pages} | {tokens} |") - return "\n".join(lines) - - -def fmt_overall(rows: list[dict], has_raw: bool) -> str: - lines = [] - if has_raw: - lines.append("| Group | N runs | Raw (total) | Raw (avg) | Consolidated (total) | Consolidated (avg) | Shrink |") - lines.append("| --- | --- | --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| **{r['group'].title()}** | {r['n']} | {r['raw_total']} | {r['raw_avg']:.1f} " - f"| {r['cons_total']} | {r['cons_avg']:.1f} | {r['shrink']:.0%} |") - else: - lines.append("| Group | N runs | Comments (total) | Comments (avg) |") - lines.append("| --- | --- | --- | --- |") - for r in rows: - lines.append(f"| **{r['group'].title()}** | {r['n']} | " - f"{r['cons_total']} | {r['cons_avg']:.1f} |") - return "\n".join(lines) - - -def fmt_per_model(rows: list[dict], has_raw: bool) -> str: - lines = [] - if has_raw: - lines.append("| Group | Model | N | Raw (total) | Raw (avg) | Cons. (total) | Cons. (avg) | Shrink |") - lines.append("| --- | --- | --- | --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| {r['group']} | {r['model']} | {r['n']} | {r['raw_total']} | {r['raw_avg']:.1f} " - f"| {r['cons_total']} | {r['cons_avg']:.1f} | {r['shrink']:.0%} |") - else: - lines.append("| Group | Model | N | Comments (total) | Comments (avg) |") - lines.append("| --- | --- | --- | --- | --- |") - for r in rows: - lines.append(f"| {r['group']} | {r['model']} | {r['n']} | " - f"{r['cons_total']} | {r['cons_avg']:.1f} |") - return "\n".join(lines) - - - -def fmt_consolidation(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Accepted shrink | Rejected shrink |") - lines.append("| --- | --- | --- |") - for r in rows: - # Use short display name - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | {r['acc_shrink']:.0%} | {r['rej_shrink']:.0%} |") - return "\n".join(lines) - - -def fmt_cost(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Accepted | Rejected | Total |") - lines.append("| --- | --- | --- | --- |") - grand_acc = grand_rej = grand_total = 0.0 - for r in rows: - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | ${r['accepted']:.2f} | ${r['rejected']:.2f} " - f"| ${r['total']:.2f} |") - grand_acc += r["accepted"] - grand_rej += r["rejected"] - grand_total += r["total"] - lines.append(f"| **Total** | **${grand_acc:.2f}** | **${grand_rej:.2f}** " - f"| **${grand_total:.2f}** |") - return "\n".join(lines) - +# (heading, ci_auc table kind) — printed in order over the discovered cells. +TABLES = [ + ("Comment volume and overall accuracy, per model", "comment_volume"), + ("Accuracy by quality proxy", "by_proxy"), + ("Accuracy by severity tier", "by_severity"), +] -def fmt_runtime(rows: list[dict]) -> str: - lines = [] - lines.append("| Model | Avg per run | Range |") - lines.append("| --- | --- | --- |") - for r in rows: - name = r["model"].split("-")[0].title() - lines.append(f"| {name} | {r['avg_min']:.1f} min " - f"| {r['min_min']:.1f}-{r['max_min']:.1f} min |") - return "\n".join(lines) +def _resolve(path: Path, base: Path) -> Path: + return path if path.is_absolute() else (base / path) -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- def main() -> None: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--config", help="YAML config file (same schema as run_study.py).") - ap.add_argument("--results-dir", help="Explicit results directory.") - ap.add_argument("--papers", action="store_true", - help="Include papers table (parses PDFs, slow).") + ap.add_argument("--config", type=Path, nargs="+", + help="one or more run config YAMLs, merged (e.g. baseline.yaml coarse.yaml)") + ap.add_argument("--results-dir", type=Path, nargs="+", + help="one or more results directories, merged") + ap.add_argument("--manifest", type=Path, + help="manifest JSON (defaults to the first config's, then manifests/canonical/full.json)") + ap.add_argument("--all-models", action="store_true", + help="show every model found, not just the manifest's roster") + ap.add_argument("--consolidated", action="store_true", + help="show openaireview's consolidated progressive output instead of the " + "pre-consolidation progressive_original the paper reports") args = ap.parse_args() - - cfg = load_config(args.config) - name = cfg.get("name") - max_pages = cfg.get("max_pages", 20) - max_tokens = cfg.get("max_tokens", 20_000) - - if args.results_dir: - results_dir = Path(args.results_dir) - elif name: - results_dir = HERE / "results" / name - else: - results_dir = HERE / "results" - - if not results_dir.exists(): - sys.exit(f"results directory not found: {results_dir}") - - manifest = json.loads((HERE / "manifest.json").read_text()) - results = load_results(results_dir, manifest) - run_log = load_run_log(results_dir) - - if not results: - sys.exit("No result files found.") - - print(f"Loaded {len(results)} result files from {results_dir}\n") - - # --- Papers table --- - pdfs_available = any((HERE / "papers" / p["group"] / f"{p['slug']}.pdf").exists() - for p in manifest["papers"]) - if args.papers and pdfs_available: - try: - print("### Papers\n") - papers_rows = compute_papers_table(manifest, max_pages, max_tokens) - print(fmt_papers(papers_rows)) - # Summary stats - acc_pages = [r["pages"] for r in papers_rows if r["group"] == "accepted" and r["pages"]] - rej_pages = [r["pages"] for r in papers_rows if r["group"] == "rejected" and r["pages"]] - acc_tok = [r["eff_tokens"] for r in papers_rows if r["group"] == "accepted" and r["eff_tokens"]] - rej_tok = [r["eff_tokens"] for r in papers_rows if r["group"] == "rejected" and r["eff_tokens"]] - if acc_pages and rej_pages: - print(f"\nAccepted avg: {sum(acc_pages)/len(acc_pages):.1f} pages, " - f"~{sum(acc_tok)/len(acc_tok):,.0f} tokens") - print(f"Rejected avg: {sum(rej_pages)/len(rej_pages):.1f} pages, " - f"~{sum(rej_tok)/len(rej_tok):,.0f} tokens") - except Exception as e: - print(f"(Papers table skipped — {e})\n", file=sys.stderr) - elif args.papers: - print("(Papers table skipped — PDFs not on disk. Run download_papers.py first.)\n") - - # --- Per-system tables --- - systems = detect_systems(results) - if not systems: - print("\n(No methods found in result JSONs.)") - for system in systems: - name = system["name"] - has_raw = system["raw_prefix"] is not None - banner = f"System: {name}" - print(f"\n\n## {banner}\n") - - overall = build_overall_table(manifest, results, system) - if overall: - print(f"\n### Overall: accepted vs rejected ({name})\n") - print(fmt_overall(overall, has_raw)) - - per_model = build_per_model_table(manifest, results, system) - if per_model: - print(f"\n### Per-model breakdown ({name})\n") - print(fmt_per_model(per_model, has_raw)) - - if has_raw: - consolidation = build_consolidation_table(manifest, results, system) - if consolidation: - print(f"\n### Consolidation effect ({name})\n") - print(fmt_consolidation(consolidation)) - - cost = build_cost_table(manifest, results, system) - if cost and any(r["total"] for r in cost): - print(f"\n### Cost ({name})\n") - print(fmt_cost(cost)) - - # --- Runtime (single table spanning all systems; derived from log) --- - if run_log: - print("\n\n## Runtime (all systems)\n") - runtime = build_runtime_table(run_log, manifest) - print(fmt_runtime(runtime)) - else: - print("\n(Runtime table skipped — no run_log.jsonl found.)") + if not args.config and not args.results_dir: + ap.error("pass --config or --results-dir") + + configs = [yaml.safe_load(p.read_text()) for p in (args.config or [])] + + dirs = [_resolve(p, Path.cwd()) for p in (args.results_dir or [])] + dirs += [STUDY / "results" / cfg["name"] for cfg in configs] + missing = [str(d) for d in dirs if not d.exists()] + if missing: + sys.exit("results directory not found: " + ", ".join(missing)) + + default_manifest = configs[0].get("manifest") if configs else None + manifest = _resolve(args.manifest or Path(default_manifest or "manifests/canonical/full.json"), STUDY) + if not manifest.exists(): + sys.exit(f"manifest not found: {manifest}") + + # Discover the (method, model) cells across the result dirs. By default keep + # only the manifest's roster models (dropping exploratory runs that linger in + # the result dirs) and the consolidated progressive output. A model-less + # system keyed as __ (e.g. reviewer3) is always kept. + slug_to_mem = load_manifest(manifest) + roster = {m.rsplit("/", 1)[-1] for m in json.loads(manifest.read_text()).get("models", [])} + cells = sorted(load_counts(dirs, set(slug_to_mem))) + if roster and not args.all_models: + cells = [c for c in cells if c[1] in roster or c[1] == c[0]] + # openaireview writes two variants per run: progressive_original (raw, the + # variant the paper reports) and progressive (after consolidation). Show the + # paper's by default; --consolidated switches to the consolidated one. + drop_variant = "progressive_original" if args.consolidated else "progressive" + cells = [c for c in cells if c[0] != drop_variant] + if not cells: + sys.exit("no (method, model) results found under: " + ", ".join(str(d) for d in dirs)) + + print(f"Results: {', '.join(d.name for d in dirs)}\nManifest: {manifest.name} " + f"({len(slug_to_mem)} papers)\nCells: {len(cells)} (method, model)\n") + for heading, kind in TABLES: + print(f"\n########## {heading} ##########") + ci_auc.DISPATCH[kind](manifest, dirs, cells) if __name__ == "__main__": diff --git a/benchmarks/conference_study/analyses/report_scaleup.py b/benchmarks/conference_study/analyses/report_scaleup.py index 79dce76..771ecb8 100644 --- a/benchmarks/conference_study/analyses/report_scaleup.py +++ b/benchmarks/conference_study/analyses/report_scaleup.py @@ -2,7 +2,7 @@ """Pair-grouped report for the 4-pair signal-matrix scale-up. Reads: - manifests/combined.json paper list + pair_memberships + word_count + manifests/canonical/full.json paper list + pair_memberships + word_count results//.json methods dict with __ keys Emits one markdown table per pair showing high-tail vs low-tail comment @@ -27,7 +27,7 @@ from statistics import mean HERE = Path(__file__).resolve().parent -MANIFEST = HERE / "manifests" / "combined.json" +MANIFEST = HERE / "manifests" / "canonical" / "full.json" RESULTS_BASE = HERE / "results" BASELINE_MANIFEST = HERE / "manifest.json" diff --git a/benchmarks/conference_study/competitors/coarse_adapter.py b/benchmarks/conference_study/competitors/coarse_adapter.py index 03634fb..7f57307 100644 --- a/benchmarks/conference_study/competitors/coarse_adapter.py +++ b/benchmarks/conference_study/competitors/coarse_adapter.py @@ -65,13 +65,14 @@ def method_key(self, model: str) -> str: return f"coarse__{model_short(model)}" def review(self, pdf: Path, model: str, cfg: dict) -> NormalizedReview: - venv_python = cfg.get("venv_python") + venv_python = cfg.get("venv_python") or os.environ.get("COARSE_VENV_PYTHON") if not venv_python: raise RuntimeError( - "coarse adapter requires `venv_python` in config " - "(path to the coarse venv's python executable)" + "coarse adapter needs the coarse venv's python: set the " + "COARSE_VENV_PYTHON env var, or `venv_python` in the config. " + "See benchmarks/conference_study/README.md for coarse setup." ) - venv_python = str(Path(venv_python).expanduser()) + venv_python = str(Path(os.path.expandvars(venv_python)).expanduser()) if not Path(venv_python).exists(): raise RuntimeError(f"coarse venv python not found: {venv_python}") diff --git a/benchmarks/conference_study/configs/baseline.yaml b/benchmarks/conference_study/configs/baseline.yaml index 8dfdb7c..d25e05e 100644 --- a/benchmarks/conference_study/configs/baseline.yaml +++ b/benchmarks/conference_study/configs/baseline.yaml @@ -1,9 +1,24 @@ -# Baseline run config for the ICLR accept-vs-reject study. +# Baseline run config for the quality-proxy study. # Results -> benchmarks/conference_study/results/baseline/ # Log -> benchmarks/conference_study/results/baseline/run_log.jsonl name: baseline +# Paper set. Regenerate the full canonical manifest with select_papers.py +# (deterministic, seed 42), or point at manifests/canonical/frontier.json +# for the 74-paper frontier subset. +manifest: manifests/canonical/full.json + +# Backbone models. These four efficient models run on the full set. The two +# frontier models (openai/gpt-5.5, anthropic/claude-opus-4.7) run only on the +# frontier subset; for that run, point `manifest` at +# manifests/canonical/frontier.json and add them here. +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + # Review caps passed to the openaireview CLI. max_pages: 20 max_tokens: 20000 diff --git a/benchmarks/conference_study/configs/coarse.yaml b/benchmarks/conference_study/configs/coarse.yaml index 258a6c4..0fdd3b4 100644 --- a/benchmarks/conference_study/configs/coarse.yaml +++ b/benchmarks/conference_study/configs/coarse.yaml @@ -4,16 +4,23 @@ # # Prerequisites: # - coarse installed in its own venv at `venv_python` below -# - OPENROUTER_API_KEY set in env (coarse routes all three manifest models +# - OPENROUTER_API_KEY set in env (coarse routes all backbone models # through OpenRouter) name: coarse competitor: coarse -# Path to the coarse venv's python. The adapter subprocess-calls a runner -# inside this venv so coarse's deps (litellm, instructor, docling, ...) stay -# isolated from openaireview's venv. -venv_python: /data/dangnguyen/openaireview_project/coarse/.venv/bin/python +# Backbone models (same efficient roster as the baseline run). +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + +# The adapter subprocess-calls a runner inside coarse's own venv so its deps +# (litellm, instructor, docling, ...) stay isolated from openaireview's. Point +# the COARSE_VENV_PYTHON env var at that venv's python (see the README section +# "Running the coarse competitor"). To hardcode it instead, set `venv_python:`. # Batch runner knobs. timeout_sec: 1800 # 30 min per (paper, model) run (coarse is 2-5 min typical) diff --git a/benchmarks/conference_study/configs/frontier.yaml b/benchmarks/conference_study/configs/frontier.yaml new file mode 100644 index 0000000..46f1bff --- /dev/null +++ b/benchmarks/conference_study/configs/frontier.yaml @@ -0,0 +1,26 @@ +# Frontier-subset run config for the quality-proxy study. +# Runs all six backbone models on the 74-paper frontier subset, the set the +# paper mainly reports on. The two frontier models (gpt-5.5, claude-opus-4.7) +# are too expensive for the full set, so they appear only here. +# The frontier manifest is not shipped; request it from the authors. +# Results -> benchmarks/conference_study/results/frontier/ + +name: frontier + +manifest: manifests/canonical/frontier.json + +models: + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + - x-ai/grok-4.1-fast + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + +# Review caps passed to the openaireview CLI. +max_pages: 20 +max_tokens: 20000 + +# Batch runner knobs. +timeout_sec: 3600 +max_per_model: 2 diff --git a/benchmarks/conference_study/download_papers.py b/benchmarks/conference_study/download_papers.py index a38891e..2db9aaf 100644 --- a/benchmarks/conference_study/download_papers.py +++ b/benchmarks/conference_study/download_papers.py @@ -7,7 +7,7 @@ on HuggingFace. Multi-venue (ICLR, NeurIPS, CoRL, etc.), uses review score thresholds instead of decisions. --source snor Scale-up 4-pair signal matrix. Reads - manifests/combined.json (produced by select_papers.py). + manifests/canonical/full.json (produced by select_papers.py). Downloads PDFs to papers/scaleup/.pdf and updates the manifest with pdf_path + pages. @@ -590,7 +590,7 @@ def main() -> None: default="papercopilot", help="Data source (default: papercopilot).") ap.add_argument("--manifest", type=Path, - help="[snor] Path to combined.json (default: manifests/combined.json).") + help="[snor] Path to the manifest (default: manifests/canonical/full.json).") ap.add_argument("--limit", type=int, help="[snor] Only download the first N manifest entries " "(for smoke tests).") @@ -615,11 +615,11 @@ def main() -> None: manifest_path = out_dir / "manifest.json" cache_dir = out_dir / ".cache" - # snor source has its own flow: read combined.json, download flat. + # snor source has its own flow: read full.json, download flat. if args.source == "snor": snor_manifest = (args.manifest if args.manifest - else out_dir / "manifests" / "v1" / "combined.json") + else out_dir / "manifests" / "canonical" / "full.json") # The papercopilot/hf default of 15 pages filters accidentally # tiny papers. SNOR already selects from real conference decisions, # so accept anything at least 6 pages. User can override explicitly. diff --git a/benchmarks/conference_study/estimate_cost.py b/benchmarks/conference_study/estimate_cost.py index 920147d..218d75a 100644 --- a/benchmarks/conference_study/estimate_cost.py +++ b/benchmarks/conference_study/estimate_cost.py @@ -73,6 +73,9 @@ "openai/gpt-5-mini": {"prompt": 0.25e-6, "completion": 2.00e-6}, "openai/gpt-5.4-mini": {"prompt": 0.75e-6, "completion": 4.50e-6}, "openai/gpt-5.5": {"prompt": 5.00e-6, "completion": 30.00e-6}, + # TODO: add the OpenRouter rate for x-ai/grok-4.1-fast (part of the efficient + # roster). Until it is filled in, estimate_cost skips Grok and the printed + # total excludes it. } diff --git a/benchmarks/conference_study/paper_metrics.py b/benchmarks/conference_study/paper_metrics.py index 461bb78..6fbda51 100644 --- a/benchmarks/conference_study/paper_metrics.py +++ b/benchmarks/conference_study/paper_metrics.py @@ -20,7 +20,7 @@ from pathlib import Path HERE = Path(__file__).resolve().parent -DEFAULT_MANIFEST = HERE / "manifests" / "combined.json" +DEFAULT_MANIFEST = HERE / "manifests" / "canonical" / "full.json" # Make the openaireview package importable when running from this dir. sys.path.insert(0, str(HERE.parent.parent / "src")) diff --git a/benchmarks/conference_study/run_study.py b/benchmarks/conference_study/run_study.py index d9a34c6..31fbc7a 100644 --- a/benchmarks/conference_study/run_study.py +++ b/benchmarks/conference_study/run_study.py @@ -235,7 +235,7 @@ def main() -> None: RESULTS_DIR = DEFAULT_RESULTS_BASE / name if name else DEFAULT_RESULTS_BASE LOG_FILE = RESULTS_DIR / "run_log.jsonl" - manifest_path = args.manifest or Path(cfg.get("manifest", "manifest.json")) + manifest_path = args.manifest or Path(cfg.get("manifest", "manifests/canonical/full.json")) if not manifest_path.is_absolute(): manifest_path = HERE / manifest_path manifest = json.loads(manifest_path.read_text()) diff --git a/benchmarks/conference_study/select_papers.py b/benchmarks/conference_study/select_papers.py index 98cd7ab..df0afd9 100644 --- a/benchmarks/conference_study/select_papers.py +++ b/benchmarks/conference_study/select_papers.py @@ -1,10 +1,10 @@ #!/usr/bin/env python """Select papers for the 4-pair signal matrix from SNOR v1. -Writes per-pair manifests to manifests/pair_{1,2,3,4}.json plus a -combined manifest (manifests/combined.json) with deduped papers and -pair-membership metadata — that combined manifest is what the downloader -and runner consume. +Writes per-pair manifests (pair_{1,2,3,4}.json) plus a full-set manifest +(full.json) with deduped papers and pair-membership metadata into the output +dir (default manifests/canonical/). full.json is what the downloader and runner +consume. Pairs: 1 top-cited (high) vs never-published (low) community-wide @@ -40,7 +40,7 @@ from snor_loader import load_cohort HERE = Path(__file__).resolve().parent -MANIFESTS_DIR = HERE / "manifests" / "v1" +MANIFESTS_DIR = HERE / "manifests" / "canonical" CURRENT_YEAR = 2026 @@ -421,18 +421,22 @@ def build_manifests( "years": years, "n_per_tail": n_per_tail, "papers": combined_papers, + # Efficient roster run on the full set. The two frontier models + # (openai/gpt-5.5, anthropic/claude-opus-4.7) run only on the frontier + # subset (manifests/canonical/frontier.json), which pins all six. "models": [ - "google/gemini-3-flash-preview", - "z-ai/glm-4.6", - "qwen/qwen3-235b-a22b-2507", + "deepseek/deepseek-v4-flash", + "qwen/qwen3.6-35b-a3b", + "google/gemini-3.1-flash-lite-preview", + "x-ai/grok-4.1-fast", ], "review_caps": {"max_pages": 20, "max_tokens": 20_000}, } - (out_dir / "combined.json").write_text( + (out_dir / "full.json").write_text( json.dumps(combined, indent=2) + "\n" ) print(f"\nCombined manifest: {len(combined_papers)} unique papers " - f"-> {out_dir / 'combined.json'}") + f"-> {out_dir / 'full.json'}") def main() -> None: From c5d85f9c83a7bb6a62abe9af1a521c170b97525c Mon Sep 17 00:00:00 2001 From: Dang Nguyen Date: Sun, 28 Jun 2026 19:24:54 -0500 Subject: [PATCH 2/5] Clean up the perturbation benchmark docs for public release Rewrite both READMEs around the per-stage run flow (perturb_automated.py for generation, then run_benchmark.py --stages prepare/review/score/report), drop the "(v2)" title, and replace the old error taxonomy with the paper's four categories (Surface / Claim / Reasoning / Experimental). Commit a canonical configs/default.yaml (configs/ was gitignored entirely) with the paper's model roster. Docs and config only; the scorer bug fix and gate/threshold scoring config land separately with #87. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 28 ++- benchmarks/perturbation/.gitignore | 5 +- benchmarks/perturbation/README.md | 176 +++++++++---------- benchmarks/perturbation/configs/default.yaml | 26 +++ 4 files changed, 125 insertions(+), 110 deletions(-) create mode 100644 benchmarks/perturbation/configs/default.yaml diff --git a/README.md b/README.md index 773c4e1..6ea61da 100644 --- a/README.md +++ b/README.md @@ -226,11 +226,11 @@ uv pip install -e ".[benchmarks]" export OPENROUTER_API_KEY=... ``` -Run scripts from inside each benchmark's directory unless noted. +Run scripts from inside each benchmark's directory unless otherwise noted. ### Quality-proxy study (`benchmarks/conference_study/`) -Tests whether AI review systems comment more on weaker papers than on stronger ones. Papers are split into high- and low-quality groups by four noisy quality proxies built from citations, venue awards, review scores, and a composite of the three. `configs/baseline.yaml` (our system) and `coarse.yaml` (coarse, an external review system) are the committed example configs. +Tests whether AI review systems comment more on weaker papers than on stronger ones. Papers are split into high- and low-quality groups by four quality proxies built from citations, venue awards, review scores, and a composite of the three. `configs/baseline.yaml` (our system) and `coarse.yaml` (coarse, an external review system) are the committed example configs. ```bash cd benchmarks/conference_study @@ -256,29 +256,25 @@ The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontie ### Perturbation benchmark (`benchmarks/perturbation/`) -Injects controlled errors (math edits, false claims, faulty reasoning, experimental flaws) into clean papers and measures per-comment recall by error type and domain. +Injects controlled errors (surface math edits, false claims, faulty reasoning, experimental flaws) into clean papers and measures recall by model, method, and error category. -Pipeline: `extract → generate → validate → verify → inject → review → score`. `run_benchmark.py` drives all stages from a single YAML. +Pipeline: `extract → generate → validate → verify → inject → review → score`. Generation is one tool, the benchmark stages another: ```bash cd benchmarks/perturbation -# One-shot: prepare papers, run reviews, score against the perturbation manifest -python run_benchmark.py configs/default.yaml +# 1. Sample papers from arXiv and generate perturbations (extract→...→inject). +python perturb_automated.py --arxiv-category "math.*" --category theoretical \ + --error-type all --target 10 --min-year 2015 -# Or run a subset of stages -python run_benchmark.py configs/default.yaml --stages prepare,review +# 2-5. Point configs/default.yaml's input_dir at step 1's output, then run the stages. +python run_benchmark.py configs/default.yaml --stages prepare +python run_benchmark.py configs/default.yaml --stages review python run_benchmark.py configs/default.yaml --stages score - -# Multi-config sweep with parallel workers reused across configs -python run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 - -# Aggregate recall tables across all (paper, model, method) cells -python generate_report.py results/ +python run_benchmark.py configs/default.yaml --stages report ``` -The config picks the review system per run via `system: openaireview | coarse | reviewer3`; adapter setup for third-party systems is in `systems/README.md`. Scoring uses a two-stage filter: a fuzzy substring match on the perturbed text against the comment quote, then an LLM judge rating (≥3/5) on whether the explanation identifies the same error. See `benchmarks/perturbation/README.md` for error-type taxonomy, results layout, and known limitations. +The config picks the review system per run via `system: openaireview | coarse | reviewer3` (adapter setup for the external systems is in `systems/README.md`). A comment counts as a detection when it passes two stages: a fuzzy substring match (the perturbed text covers the comment quote), then an LLM judge rating (a 1-5 score >= 3) on whether the explanation identifies the same error. See `benchmarks/perturbation/README.md` for the error taxonomy, config schema, and results layout. ## Related Resources diff --git a/benchmarks/perturbation/.gitignore b/benchmarks/perturbation/.gitignore index 5cce8cb..17108f1 100644 --- a/benchmarks/perturbation/.gitignore +++ b/benchmarks/perturbation/.gitignore @@ -9,8 +9,9 @@ data/ # Reports (regeneratable from results/) reports/ -# Local experiment configs (we don't check any of these in). -configs/ +# Experiment configs are local except the committed example. +configs/* +!configs/default.yaml # Python __pycache__/ diff --git a/benchmarks/perturbation/README.md b/benchmarks/perturbation/README.md index a9521f3..69e76c1 100644 --- a/benchmarks/perturbation/README.md +++ b/benchmarks/perturbation/README.md @@ -1,126 +1,118 @@ -# Perturbation Benchmark (v2) +# Perturbation Benchmark -Benchmark for evaluating how well LLM reviewers detect **seeded errors** in mathematical papers. The pipeline takes real arxiv papers, injects controlled perturbations, runs automated reviews, and measures recall. +Do AI review systems catch errors deliberately injected into otherwise-good +papers? This benchmark takes real arXiv papers, injects controlled errors, runs +automated reviews, and measures recall (the fraction of injected errors a system +detects). + +The paper has the full method and results +([arXiv:2606.19749](https://arxiv.org/abs/2606.19749)). This README covers how to +reproduce a run. ## Pipeline ``` -extract → generate → validate → inject → review → score +extract → generate → validate → verify → inject → review → score ``` -1. **Extract** (`extract.py`): Identify candidate math spans (equations, symbols) in the paper. -2. **Generate** (`generate.py`): Use an LLM to create perturbations for each candidate span. -3. **Validate** (`validate.py`): Check that perturbations are valid (original exists, no overlaps, no garbled text). -4. **Inject** (`inject.py`): Apply valid perturbations to produce a corrupted paper. -5. **Review**: Run `openaireview review` on the corrupted paper with each (model, method) combination. -6. **Score** (`score.py`): Compare review comments against the perturbation manifest using fuzzy substring matching + LLM-as-judge. +- **extract → generate → validate → verify → inject** (the `openaireview perturb` + CLI, driven by `perturb_automated.py`): find candidate spans in a paper, + generate candidate errors, validate and verify them, and inject the kept ones + to produce a corrupted paper plus a ground-truth manifest. +- **review** (`run_benchmark.py`): run each (model, method) over the corrupted paper. +- **score** (`run_benchmark.py`): match each review comment against the manifest + and compute recall. -## Error Types +## Error types -### Surface errors -Minimal single-token edits to math expressions: -- `operator_or_sign` — flip `+`/`-`, `≤`/`≥`, `∪`/`∩` -- `symbol_binding` — swap a symbol (`α`→`β`) -- `index_or_subscript` — change sub/superscript (`x_i`→`x_{i+1}`) -- `numeric_parameter` — change a number (`0.5`→`0.25`) +Injected errors span four categories (the underlying error type is in parentheses): -### Formal errors -Deeper structural corruptions to definitions, theorems, and proofs: -- `def_wrong`, `thm_wrong_condition`, `thm_wrong_conclusion`, `thm_wrong_scope` -- `proof_wrong_direction`, `proof_missing_case`, `proof_wrong_assumption`, `proof_mismatch` +- **Surface** (`surface`): minimal single-token math edits (flip an operator or sign, change a number, alter a subscript). +- **Claim** (`statement_empirical`, `claim_theoretical`): a stated empirical or theoretical claim is made wrong. +- **Reasoning** (`logic`): a logical or derivation step is broken. +- **Experimental** (`experimental`): an experimental-design or setup detail is corrupted. -## Quick Start +## Quick start -```bash -# Install benchmark dependencies -pip install -e ".[benchmarks]" +Install the benchmark dependencies and set an API key: -# Run a single config (prepare → review → score → report) -python benchmarks/perturbation/run_benchmark.py benchmarks/perturbation/configs/default.yaml +```bash +uv pip install -e ".[benchmarks]" +export OPENROUTER_API_KEY=... +cd benchmarks/perturbation +``` -# Run only specific stages -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages prepare,review -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages score +Run the pipeline one stage at a time: -# Run many domains in one process (workers stay busy across config boundaries) -python benchmarks/perturbation/run_benchmark.py --configs configs/full_*.yaml \ - --parallel-openaireview 2 --parallel-coarse 8 +```bash +# 1. Sample papers from arXiv and generate perturbations (extract→...→inject). +# Writes corrupted papers + manifests under results/perturbations///. +python perturb_automated.py --arxiv-category "math.*" --category theoretical \ + --error-type all --target 10 --min-year 2015 + +# Point the config's input_dir at step 1's output, then run the benchmark stages: +python run_benchmark.py configs/default.yaml --stages prepare # 2. stage corrupted papers +python run_benchmark.py configs/default.yaml --stages review # 3. run the reviews +python run_benchmark.py configs/default.yaml --stages score # 4. score against the manifests +python run_benchmark.py configs/default.yaml --stages report # 5. aggregate the recall tables ``` -`run_benchmark.py` selects a review system per config via the `system:` field -(`openaireview` | `coarse` | `reviewer3`); see `systems/README.md` for setup -of the third-party systems. - -## Configuration +Run several stages at once with `--stages prepare,review,score,report`, or sweep +many configs with `--configs configs/*.yaml`. `run_benchmark.py` selects the +review system per config via the `system:` field (`openaireview` | `coarse` | +`reviewer3`). See `systems/README.md` for setting up the external systems. -Configs are YAML files in `configs/`. Copy `default.yaml` and edit to create experiment variants. Committed configs serve as the experiment log. +## Config schema ```yaml -max_papers: 5 -length: short # short (2k-7k words) | medium (7k-17k) | long (>17k) -error_type: surface # surface | formal | all -score_method: llm # llm | fuzzy | semantic - -perturb_model: google/gemini-3-flash-preview -score_model: google/gemini-3-flash-preview - -review_models: - - google/gemini-3-flash-preview - - z-ai/glm-4.6 - -review_methods: +system: openaireview # openaireview | coarse | reviewer3 +input_dir: results/perturbations/math_all # corrupted papers from perturb_automated.py +results_dir: results/default # staged papers, reviews, scores + +models: # backbone models to review with + - openai/gpt-5.5 + - deepseek/deepseek-v4-flash +methods: # zero_shot | local | progressive - zero_shot - progressive -results_dir: benchmarks/perturbation/results +score_method: llm # llm | fuzzy | semantic +score_model: google/gemini-3-flash-preview # the llm judge ``` -Papers are streamed from the [proof-pile](https://huggingface.co/datasets/hoskinson-center/proof-pile) dataset and binned by word count. +Configs in `configs/` are local except the committed `default.yaml`. Copy it to +define experiment variants. -## Results Layout - -``` -/ - config.yaml # resolved config - perturb//paper_001/ - paper_001.md # original paper - paper-001_clean.md # clean copy - paper-001_corrupted.md # with injected errors - paper-001_perturbations.json # ground-truth manifest - ///paper_001/ - review/*.json # review results - score//*_score.json # recall scores -``` +## Scoring -## Reports +A comment counts as detecting an injected error when it passes two stages: -Experiment reports are in `reports/`. See `reports/surface_3models_short_medium.md` for the first benchmark run (3 cheap models, 10 papers, surface errors). +1. **Substring match** — the perturbed text approximately covers the comment's + quote (normalized, 0.75 coverage), so the judge only sees comments pointing at + the right span. +2. **LLM judge** — Gemini-3 Flash Preview rates whether the comment's explanation + identifies the same error as the manifest's `why_wrong`, on a 1-5 scale, and a + rating >= 3 counts as detected. -### Generating report statistics +Recall is the fraction of injected errors detected, reported per (model, method) +and per error category by the `report` stage. -After a pipeline run completes, use `generate_report.py` to aggregate results across all papers, models, and methods: +## Results layout -```bash -python benchmarks/perturbation/generate_report.py benchmarks/perturbation/results_short +``` +/ + perturb//paper_NNN/ + paper_NNN_corrupted.md # injected paper + paper_NNN_perturbations.json # ground-truth manifest + ///paper_NNN/ + review/*.json # review output + score//*_score.json # recall scores ``` -This prints markdown tables to stdout covering: -- Configuration summary -- Ground truth counts by length and error type -- Recall by model × method (split by paper length and overall) -- Recall by error type -- Token usage and cost - -The output is meant to be reviewed and edited into a final report in `reports/`. - -## Scoring - -The scorer uses a two-stage filter: -1. **Fuzzy substring match** — checks if the perturbed text appears (approximately) in the review comment's quote, using normalized text coverage with a 0.75 threshold. -2. **LLM-as-judge** — asks a model to rate whether the reviewer's explanation identifies the same error described in the perturbation's `why_wrong` field (score >= 3/5 = match). - -## Known Limitations +## Known limitations -- The perturb stage targets `n_per_error=2` perturbations per error type (8 total per paper), but the LLM often reuses the same candidate span for both, causing the validator to reject the duplicate. Typical yield is ~4-5 per paper. -- Fuzzy substring matching can miss catches where the reviewer heavily paraphrases the quoted text. -- `cost_usd` from OpenRouter metadata is unreliable for some models (notably qwen). +- The generator often reuses the same candidate span for multiple errors, and the + validator rejects the duplicates, so typical yield is ~4-5 perturbations per paper. +- The substring match can miss detections where the reviewer heavily paraphrases + the quoted text. +- `cost_usd` from OpenRouter metadata is unreliable for some models. diff --git a/benchmarks/perturbation/configs/default.yaml b/benchmarks/perturbation/configs/default.yaml new file mode 100644 index 0000000..cd6ba8a --- /dev/null +++ b/benchmarks/perturbation/configs/default.yaml @@ -0,0 +1,26 @@ +# Default perturbation benchmark config (one experiment = one config). +# +# input_dir: corrupted papers + ground-truth manifests from perturb_automated.py +# (its output dir, e.g. results/perturbations//). +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: results/perturbations/math_all +results_dir: results/default + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview From 080d4f0ca073e66bfa32964358831dbce0520556 Mon Sep 17 00:00:00 2001 From: Dang Nguyen Date: Mon, 29 Jun 2026 00:47:05 -0500 Subject: [PATCH 3/5] Perturbation benchmark: ship released set + per-domain configs, demote verify Make the perturbation benchmark reproducible from a fresh clone: - Add a Dataset section pointing to the released perturbed papers (zip), and make the released set the recommended reproduction path. - Add eight per-domain configs (full_*.yaml) so `--configs configs/full_*.yaml` reproduces the full 74-paper benchmark in one command; whitelist them in .gitignore. - Fix input_dir to the /all level (the layout discover_units expects), in default.yaml and the README example; the old results/perturbations path resolved nowhere. - Document the real generation chain (perturb_automated -> verify_existing -> reinject_existing) with correct, repo-root-aware commands. - Demote the LLM verify step to the optional regeneration path (docs only), with a note that the released set was verifier-filtered and manually audited. - gitignore the repo-root results/ that run_benchmark.py writes to. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 + README.md | 19 ++- benchmarks/perturbation/.gitignore | 4 +- benchmarks/perturbation/README.md | 108 +++++++++++++----- benchmarks/perturbation/configs/default.yaml | 6 +- .../perturbation/configs/full_cs_CC.yaml | 27 +++++ .../perturbation/configs/full_cs_LG.yaml | 27 +++++ .../perturbation/configs/full_econ_EM.yaml | 27 +++++ .../perturbation/configs/full_hep-ex.yaml | 27 +++++ .../perturbation/configs/full_math_all.yaml | 27 +++++ .../configs/full_physics_atm-clus.yaml | 27 +++++ .../perturbation/configs/full_q-bio_GN.yaml | 27 +++++ .../perturbation/configs/full_stat_AP.yaml | 27 +++++ 13 files changed, 308 insertions(+), 47 deletions(-) create mode 100644 benchmarks/perturbation/configs/full_cs_CC.yaml create mode 100644 benchmarks/perturbation/configs/full_cs_LG.yaml create mode 100644 benchmarks/perturbation/configs/full_econ_EM.yaml create mode 100644 benchmarks/perturbation/configs/full_hep-ex.yaml create mode 100644 benchmarks/perturbation/configs/full_math_all.yaml create mode 100644 benchmarks/perturbation/configs/full_physics_atm-clus.yaml create mode 100644 benchmarks/perturbation/configs/full_q-bio_GN.yaml create mode 100644 benchmarks/perturbation/configs/full_stat_AP.yaml diff --git a/.gitignore b/.gitignore index ea7263f..542eac7 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,8 @@ venv/ review_results/ benchmarks/conference_study/results/ benchmarks/perturbation/perturbation_results/ +# Perturbation benchmark run outputs (run_benchmark.py resolves results_dir here) +/results/ # conference_study study artifacts (not code). The full paper set regenerates # via select_papers.py, so manifests/ is ignored. The frontier subset has no diff --git a/README.md b/README.md index 6ea61da..68a4b97 100644 --- a/README.md +++ b/README.md @@ -258,23 +258,18 @@ The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontie Injects controlled errors (surface math edits, false claims, faulty reasoning, experimental flaws) into clean papers and measures recall by model, method, and error category. -Pipeline: `extract → generate → validate → verify → inject → review → score`. Generation is one tool, the benchmark stages another: +Pipeline: `extract → generate → validate → verify → inject → review → score`. + +We ship the perturbed papers used in the paper, so the generation pipeline (everything through inject) is optional. The recommended way to reproduce is to download and unzip the released set (TODO: add the Google Drive link) inside `benchmarks/perturbation/` so it lands at `data/perturbations_filtered/`, then run every paper domain at once with the committed per-domain configs: ```bash cd benchmarks/perturbation - -# 1. Sample papers from arXiv and generate perturbations (extract→...→inject). -python perturb_automated.py --arxiv-category "math.*" --category theoretical \ - --error-type all --target 10 --min-year 2015 - -# 2-5. Point configs/default.yaml's input_dir at step 1's output, then run the stages. -python run_benchmark.py configs/default.yaml --stages prepare -python run_benchmark.py configs/default.yaml --stages review -python run_benchmark.py configs/default.yaml --stages score -python run_benchmark.py configs/default.yaml --stages report +python run_benchmark.py --configs configs/full_*.yaml --stages prepare,review,score,report ``` -The config picks the review system per run via `system: openaireview | coarse | reviewer3` (adapter setup for the external systems is in `systems/README.md`). A comment counts as a detection when it passes two stages: a fuzzy substring match (the perturbed text covers the comment quote), then an LLM judge rating (a 1-5 score >= 3) on whether the explanation identifies the same error. See `benchmarks/perturbation/README.md` for the error taxonomy, config schema, and results layout. +The config picks the review system per run via `system: openaireview | coarse | reviewer3` (adapter setup for the external systems is in `systems/README.md`). A comment counts as a detection when it passes two stages: a fuzzy substring match (the perturbed text covers the comment quote), then an LLM judge rating (a 1-5 score >= 3) on whether the explanation identifies the same error. + +To regenerate a fresh perturbed set instead, see `benchmarks/perturbation/README.md`, which documents the full generation pipeline, error taxonomy, config schema, dataset layout, and results layout. ## Related Resources diff --git a/benchmarks/perturbation/.gitignore b/benchmarks/perturbation/.gitignore index 17108f1..db8465e 100644 --- a/benchmarks/perturbation/.gitignore +++ b/benchmarks/perturbation/.gitignore @@ -9,9 +9,11 @@ data/ # Reports (regeneratable from results/) reports/ -# Experiment configs are local except the committed example. +# Experiment configs are local except the committed example and the per-domain +# configs that reproduce the paper (run with --configs configs/full_*.yaml). configs/* !configs/default.yaml +!configs/full_*.yaml # Python __pycache__/ diff --git a/benchmarks/perturbation/README.md b/benchmarks/perturbation/README.md index 69e76c1..6a5235b 100644 --- a/benchmarks/perturbation/README.md +++ b/benchmarks/perturbation/README.md @@ -5,8 +5,7 @@ papers? This benchmark takes real arXiv papers, injects controlled errors, runs automated reviews, and measures recall (the fraction of injected errors a system detects). -The paper has the full method and results -([arXiv:2606.19749](https://arxiv.org/abs/2606.19749)). This README covers how to +See the paper for full method and results ([arXiv:2606.19749](https://arxiv.org/abs/2606.19749)). This README covers how to reproduce a run. ## Pipeline @@ -15,13 +14,43 @@ reproduce a run. extract → generate → validate → verify → inject → review → score ``` -- **extract → generate → validate → verify → inject** (the `openaireview perturb` - CLI, driven by `perturb_automated.py`): find candidate spans in a paper, - generate candidate errors, validate and verify them, and inject the kept ones - to produce a corrupted paper plus a ground-truth manifest. +- **extract → generate → validate** (`perturb_automated.py`, the `openaireview perturb` CLI): find candidate spans in a paper, generate candidate errors, and +apply the structural checks, writing a ground-truth manifest per paper. +- **verify** (`analyses/verify_existing.py`): rate each surviving error and drop +the ones that read as typos or do not constitute a real error. +- **inject** (`reinject_existing.py`): apply the kept errors to the clean source, +writing a corrupted paper (`*_recorrupted.md`) plus its filtered manifest +(`*_kept_perturbations.json`). These files are the input to the review stage. - **review** (`run_benchmark.py`): run each (model, method) over the corrupted paper. - **score** (`run_benchmark.py`): match each review comment against the manifest - and compute recall. +and compute recall. + +## Dataset + +The recommended way to reproduce the paper is to use the released perturbed +papers, so most users can skip generation entirely. The exact set is published as +a zip archive (TODO: add the Google Drive link). Each paper +ships as a corrupted markdown file (`*_recorrupted.md`) and a ground-truth +manifest (`*_kept_perturbations.json`), laid out as +`/all///`. Unzip it inside `benchmarks/perturbation/` +so it lands at `data/perturbations_filtered/`, then run every domain at once with +the committed per-domain configs (each `full_*.yaml` already points `input_dir` at +its split): + +```bash +cd benchmarks/perturbation +# unzip the released archive here first (creates data/perturbations_filtered/) +python run_benchmark.py --configs configs/full_*.yaml --stages prepare,review,score,report +``` + +For a single domain, pass one config instead, for example +`run_benchmark.py configs/full_math_all.yaml --stages ...`. + +The released set was filtered with an LLM verifier (the verify stage below) and +spot-checked by a manual audit (82.5% of a 40-perturbation sample judged valid +errors, see the paper). To build a fresh set of your own instead, follow the +Quick start below. Error generation is model-driven, so a fresh run produces +different papers and errors from the released set. ## Error types @@ -39,34 +68,57 @@ Install the benchmark dependencies and set an API key: ```bash uv pip install -e ".[benchmarks]" export OPENROUTER_API_KEY=... -cd benchmarks/perturbation ``` -Run the pipeline one stage at a time: +This rebuilds the perturbed papers from scratch. Most users do not need it: the +released set (see Dataset above) is the recommended input. Regenerate only to +make a fresh set. Steps 1-3 build the perturbed papers (the generation pipeline), +steps 4-7 run the benchmark over them. Step 2, verify, is an LLM filter that drops +non-substantive perturbations; it is how the released set was built, and inject +keeps only the perturbations it marks substantive. The verify and inject steps are +package modules, so they run from the repo root (the `(cd ../.. && ...)` subshells +below); the others run from this directory. ```bash -# 1. Sample papers from arXiv and generate perturbations (extract→...→inject). -# Writes corrupted papers + manifests under results/perturbations///. -python perturb_automated.py --arxiv-category "math.*" --category theoretical \ - --error-type all --target 10 --min-year 2015 +cd benchmarks/perturbation -# Point the config's input_dir at step 1's output, then run the benchmark stages: -python run_benchmark.py configs/default.yaml --stages prepare # 2. stage corrupted papers -python run_benchmark.py configs/default.yaml --stages review # 3. run the reviews -python run_benchmark.py configs/default.yaml --stages score # 4. score against the manifests -python run_benchmark.py configs/default.yaml --stages report # 5. aggregate the recall tables +# 1. extract → generate → validate: sample papers and write a clean copy plus a +# candidate manifest (_clean.md + _perturbations.json) per paper. Generation +# is model-driven, so this produces a fresh set each run. +python perturb_automated.py --arxiv-category "math.*" --category theoretical \ + --error-type all --target 10 --min-year 2015 \ + --output-dir data/perturbations/math_all + +# 2. verify: rate each candidate error and record verdicts under +# results/error_verification/. +(cd ../.. && python -m benchmarks.perturbation.analyses.verify_existing) + +# 3. inject: apply the kept (substantive) errors to the clean source, writing the +# corrupted papers (_recorrupted.md) and filtered manifests +# (_kept_perturbations.json). This is the benchmark input. +(cd ../.. && python -m benchmarks.perturbation.reinject_existing \ + --output-root benchmarks/perturbation/data/perturbations_filtered) + +# 4-7. the config's input_dir already points at the injected set +# (data/perturbations_filtered/math_all/all); run the benchmark stages. +python run_benchmark.py configs/default.yaml --stages prepare # 4. stage corrupted papers +python run_benchmark.py configs/default.yaml --stages review # 5. run the reviews +python run_benchmark.py configs/default.yaml --stages score # 6. score against the manifests +python run_benchmark.py configs/default.yaml --stages report # 7. aggregate the recall tables ``` Run several stages at once with `--stages prepare,review,score,report`, or sweep many configs with `--configs configs/*.yaml`. `run_benchmark.py` selects the review system per config via the `system:` field (`openaireview` | `coarse` | -`reviewer3`). See `systems/README.md` for setting up the external systems. +`reviewer3`). See `systems/README.md` for setting up the external systems. To skip +generation entirely, download the released set (see Dataset above) and point +`input_dir` at it. ## Config schema ```yaml system: openaireview # openaireview | coarse | reviewer3 -input_dir: results/perturbations/math_all # corrupted papers from perturb_automated.py +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all # injected papers from reinject_existing.py (or the released set) results_dir: results/default # staged papers, reviews, scores models: # backbone models to review with @@ -80,18 +132,19 @@ score_method: llm # llm | fuzzy | semantic score_model: google/gemini-3-flash-preview # the llm judge ``` -Configs in `configs/` are local except the committed `default.yaml`. Copy it to -define experiment variants. +Configs in `configs/` are local except the committed `default.yaml` and the +per-domain `full_*.yaml` set (one per paper domain), which reproduce the paper via +`--configs configs/full_*.yaml`. Copy any of them to define experiment variants. ## Scoring A comment counts as detecting an injected error when it passes two stages: 1. **Substring match** — the perturbed text approximately covers the comment's - quote (normalized, 0.75 coverage), so the judge only sees comments pointing at + quote (normalized, 0.75 coverage), so the judge only sees comments pointing at the right span. 2. **LLM judge** — Gemini-3 Flash Preview rates whether the comment's explanation - identifies the same error as the manifest's `why_wrong`, on a 1-5 scale, and a + identifies the same error as the manifest's `why_wrong`, on a 1-5 scale, and a rating >= 3 counts as detected. Recall is the fraction of injected errors detected, reported per (model, method) @@ -109,10 +162,3 @@ and per error category by the `report` stage. score//*_score.json # recall scores ``` -## Known limitations - -- The generator often reuses the same candidate span for multiple errors, and the - validator rejects the duplicates, so typical yield is ~4-5 perturbations per paper. -- The substring match can miss detections where the reviewer heavily paraphrases - the quoted text. -- `cost_usd` from OpenRouter metadata is unreliable for some models. diff --git a/benchmarks/perturbation/configs/default.yaml b/benchmarks/perturbation/configs/default.yaml index cd6ba8a..2442be3 100644 --- a/benchmarks/perturbation/configs/default.yaml +++ b/benchmarks/perturbation/configs/default.yaml @@ -1,10 +1,10 @@ # Default perturbation benchmark config (one experiment = one config). # -# input_dir: corrupted papers + ground-truth manifests from perturb_automated.py -# (its output dir, e.g. results/perturbations//). +# input_dir: injected papers + filtered manifests from reinject_existing.py, or +# the released dataset. Resolved relative to the repo root. # results_dir: where staged papers, reviews, and scores land. system: openaireview -input_dir: results/perturbations/math_all +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all results_dir: results/default # Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. diff --git a/benchmarks/perturbation/configs/full_cs_CC.yaml b/benchmarks/perturbation/configs/full_cs_CC.yaml new file mode 100644 index 0000000..44d2f84 --- /dev/null +++ b/benchmarks/perturbation/configs/full_cs_CC.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the cs_CC split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/cs_CC/all +results_dir: results/full_cs_CC + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_cs_LG.yaml b/benchmarks/perturbation/configs/full_cs_LG.yaml new file mode 100644 index 0000000..5c82dc9 --- /dev/null +++ b/benchmarks/perturbation/configs/full_cs_LG.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the cs_LG split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/cs_LG/all +results_dir: results/full_cs_LG + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_econ_EM.yaml b/benchmarks/perturbation/configs/full_econ_EM.yaml new file mode 100644 index 0000000..43e50ba --- /dev/null +++ b/benchmarks/perturbation/configs/full_econ_EM.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the econ_EM split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/econ_EM/all +results_dir: results/full_econ_EM + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_hep-ex.yaml b/benchmarks/perturbation/configs/full_hep-ex.yaml new file mode 100644 index 0000000..931588e --- /dev/null +++ b/benchmarks/perturbation/configs/full_hep-ex.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the hep-ex split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/hep-ex/all +results_dir: results/full_hep-ex + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_math_all.yaml b/benchmarks/perturbation/configs/full_math_all.yaml new file mode 100644 index 0000000..0ba8d48 --- /dev/null +++ b/benchmarks/perturbation/configs/full_math_all.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the math_all split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/math_all/all +results_dir: results/full_math_all + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_physics_atm-clus.yaml b/benchmarks/perturbation/configs/full_physics_atm-clus.yaml new file mode 100644 index 0000000..6987c8a --- /dev/null +++ b/benchmarks/perturbation/configs/full_physics_atm-clus.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the physics_atm-clus split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/physics_atm-clus/all +results_dir: results/full_physics_atm-clus + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_q-bio_GN.yaml b/benchmarks/perturbation/configs/full_q-bio_GN.yaml new file mode 100644 index 0000000..e30faf0 --- /dev/null +++ b/benchmarks/perturbation/configs/full_q-bio_GN.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the q-bio_GN split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/q-bio_GN/all +results_dir: results/full_q-bio_GN + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview diff --git a/benchmarks/perturbation/configs/full_stat_AP.yaml b/benchmarks/perturbation/configs/full_stat_AP.yaml new file mode 100644 index 0000000..328e95c --- /dev/null +++ b/benchmarks/perturbation/configs/full_stat_AP.yaml @@ -0,0 +1,27 @@ +# Per-domain perturbation benchmark config for the stat_AP split. +# Run every domain at once: run_benchmark.py --configs configs/full_*.yaml +# +# input_dir: injected papers + filtered manifests (the released set), resolved +# relative to the repo root. +# results_dir: where staged papers, reviews, and scores land. +system: openaireview +input_dir: benchmarks/perturbation/data/perturbations_filtered/stat_AP/all +results_dir: results/full_stat_AP + +# Backbone models reviewed (the paper's six). Trim for a cheaper smoke run. +models: + - openai/gpt-5.5 + - anthropic/claude-opus-4.7 + - x-ai/grok-4.1-fast + - deepseek/deepseek-v4-flash + - qwen/qwen3.6-35b-a3b + - google/gemini-3.1-flash-lite-preview + +methods: + - zero_shot + - progressive + +# Scoring: an LLM judge (Gemini-3 Flash Preview) rates each comment that passes a +# fuzzy substring match against the perturbed text. +score_method: llm +score_model: google/gemini-3-flash-preview From 4c5c1786da1f057de936b2c9bc9cbbb883253d27 Mon Sep 17 00:00:00 2001 From: Dang Nguyen Date: Mon, 29 Jun 2026 00:50:12 -0500 Subject: [PATCH 4/5] Add Google Drive link for the released perturbation set Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- benchmarks/perturbation/README.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 68a4b97..12e507e 100644 --- a/README.md +++ b/README.md @@ -260,7 +260,7 @@ Injects controlled errors (surface math edits, false claims, faulty reasoning, e Pipeline: `extract → generate → validate → verify → inject → review → score`. -We ship the perturbed papers used in the paper, so the generation pipeline (everything through inject) is optional. The recommended way to reproduce is to download and unzip the released set (TODO: add the Google Drive link) inside `benchmarks/perturbation/` so it lands at `data/perturbations_filtered/`, then run every paper domain at once with the committed per-domain configs: +We ship the perturbed papers used in the paper, so the generation pipeline (everything through inject) is optional. The recommended way to reproduce is to download and unzip the released set ([Google Drive](https://drive.google.com/file/d/10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3/view?usp=sharing), or `gdown 10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3`) inside `benchmarks/perturbation/` so it lands at `data/perturbations_filtered/`, then run every paper domain at once with the committed per-domain configs: ```bash cd benchmarks/perturbation diff --git a/benchmarks/perturbation/README.md b/benchmarks/perturbation/README.md index 6a5235b..7e9aa8c 100644 --- a/benchmarks/perturbation/README.md +++ b/benchmarks/perturbation/README.md @@ -29,7 +29,9 @@ and compute recall. The recommended way to reproduce the paper is to use the released perturbed papers, so most users can skip generation entirely. The exact set is published as -a zip archive (TODO: add the Google Drive link). Each paper +a zip archive on +[Google Drive](https://drive.google.com/file/d/10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3/view?usp=sharing) +(download with `gdown 10tI0prXCDtyBFv_gdHFlU0i7SWN3zMr3`). Each paper ships as a corrupted markdown file (`*_recorrupted.md`) and a ground-truth manifest (`*_kept_perturbations.json`), laid out as `/all///`. Unzip it inside `benchmarks/perturbation/` From 2c1f4fe13af2e0ecfde9e222025f494db6339d75 Mon Sep 17 00:00:00 2001 From: Dang Nguyen Date: Mon, 29 Jun 2026 00:59:07 -0500 Subject: [PATCH 5/5] Publish quality-proxy manifests via Google Drive Replace the contact-the-authors step for the conference (quality-proxy) study with a Google Drive download. The zip bundles manifests/canonical/full.json and frontier.json (~274K); unzip inside benchmarks/conference_study/. full.json still regenerates via select_papers.py. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 2 +- README.md | 2 +- benchmarks/conference_study/README.md | 13 ++++++++----- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/.gitignore b/.gitignore index 542eac7..90691e1 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,7 @@ benchmarks/perturbation/perturbation_results/ # conference_study study artifacts (not code). The full paper set regenerates # via select_papers.py, so manifests/ is ignored. The frontier subset has no -# regeneration script and is not shipped (request it from the authors). +# regeneration script; it is published as a Google Drive zip (see the READMEs). benchmarks/conference_study/manifests/ benchmarks/conference_study/reports/ # Symlink targets — trailing-slash patterns wouldn't match the symlinks diff --git a/README.md b/README.md index 12e507e..d7df253 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,7 @@ python run_competitors.py --config configs/coarse.yaml python analyses/generate_report.py --config configs/baseline.yaml ``` -The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontier subset the paper mainly reports on has no regeneration script and is not shipped; contact the authors for its manifest to reproduce those results. `run_study.py` and `run_competitors.py` are idempotent, so rerunning skips paper × model combos already complete. See `benchmarks/conference_study/README.md` for more details, include how to run `coarse`. +The full 197-paper set regenerates from `select_papers.py`. The 74-paper frontier subset the paper mainly reports on has no regeneration script; download it (bundled with the full manifest) from [Google Drive](https://drive.google.com/file/d/1mor434X2tTJKBYsNXgdpxPW7guzqbJAV/view?usp=sharing) (or `gdown 1mor434X2tTJKBYsNXgdpxPW7guzqbJAV`) and unzip inside `benchmarks/conference_study/`. `run_study.py` and `run_competitors.py` are idempotent, so rerunning skips paper × model combos already complete. See `benchmarks/conference_study/README.md` for more details, include how to run `coarse`. ### Perturbation benchmark (`benchmarks/perturbation/`) diff --git a/benchmarks/conference_study/README.md b/benchmarks/conference_study/README.md index b2b5ce0..51052c5 100644 --- a/benchmarks/conference_study/README.md +++ b/benchmarks/conference_study/README.md @@ -48,9 +48,11 @@ Selection is deterministic (fixed seed, forum-id tiebreak), so this reproduces the exact 197-paper set used in the paper. **Frontier subset (74 papers)** is a fixed subsample of the full set with no -regeneration script, so it is not shipped here. The pipeline runs end to end on -the full set with the repo code; to reproduce the paper's frontier-subset -numbers, contact the authors for `manifests/canonical/frontier.json`. +regeneration script. Download it (bundled with the full manifest) from +[Google Drive](https://drive.google.com/file/d/1mor434X2tTJKBYsNXgdpxPW7guzqbJAV/view?usp=sharing) +(or `gdown 1mor434X2tTJKBYsNXgdpxPW7guzqbJAV`) and unzip inside `conference_study/` +so the files land at `manifests/canonical/`. The same archive includes +`full.json`, so you can use it to skip the SNOR download above. ## Run flow @@ -82,8 +84,9 @@ pipeline end to end for a few cents before launching the full set. Both runners are idempotent: rerunning skips (paper, model) combos already complete. To reproduce the paper's frontier-subset numbers (all six models, including the -two frontier ones), obtain `manifests/canonical/frontier.json` from the authors, -then run steps 4 and 5 with `configs/frontier.yaml` in place of `baseline.yaml`. +two frontier ones), use `manifests/canonical/frontier.json` from the download +above, then run steps 4 and 5 with `configs/frontier.yaml` in place of +`baseline.yaml`. ## Running coarse