Motivation
OpenAIReview's recall is uneven across paper domains. On the full 72-paper benchmark (GPT-5.5 backend), per-domain recall ranges from 91.8% (Econometrics) down to 45.7% (Complexity), with theoretical domains lagging empirical-science domains by a wide margin:
| Domain |
OpenAIReview recall |
| Econometrics |
91.8% |
| ML |
84.7% |
| Atomic Phys. |
72.0% |
| HEP |
71.8% |
| Genomics |
71.3% |
| Statistics |
66.1% |
| Math |
62.1% |
| Complexity |
45.7% |
The ~46-point spread suggests the pipeline's prompts, chunking, or section-routing are tuned implicitly toward some domains and underperform on others — particularly theoretical/proof-heavy papers (Math, Complexity).
Goal
Add domain calibration: adjust the review pipeline so per-domain recall is more uniform, without regressing on the strong domains.
Possible directions (open to contributors):
- Domain-specific prompts or sub-agent templates (e.g. proof-checking emphasis for Math/Complexity).
- Domain-aware section routing or chunk weighting.
- Detect domain up-front (from arXiv category, abstract, or content) and dispatch a tailored pipeline.
- Calibrate the filter step's threshold per domain.
Testing requirement
This change touches the core pipeline, so we need proper per-domain regression testing before merging.
Where to start
- Pipeline:
src/reviewer/method_progressive.py, method_zero_shot.py, method_local.py.
- Skill prompts:
src/reviewer/skill/references/subagent_templates.md, criteria.md.
- Benchmark runner:
benchmarks/perturbation/run_benchmark.py.
- Evaluation:
benchmarks/perturbation/score.py.
Happy to discuss design before implementation — drop ideas in this thread.
Motivation
OpenAIReview's recall is uneven across paper domains. On the full 72-paper benchmark (GPT-5.5 backend), per-domain recall ranges from 91.8% (Econometrics) down to 45.7% (Complexity), with theoretical domains lagging empirical-science domains by a wide margin:
The ~46-point spread suggests the pipeline's prompts, chunking, or section-routing are tuned implicitly toward some domains and underperform on others — particularly theoretical/proof-heavy papers (Math, Complexity).
Goal
Add domain calibration: adjust the review pipeline so per-domain recall is more uniform, without regressing on the strong domains.
Possible directions (open to contributors):
Testing requirement
This change touches the core pipeline, so we need proper per-domain regression testing before merging.
Where to start
src/reviewer/method_progressive.py,method_zero_shot.py,method_local.py.src/reviewer/skill/references/subagent_templates.md,criteria.md.benchmarks/perturbation/run_benchmark.py.benchmarks/perturbation/score.py.Happy to discuss design before implementation — drop ideas in this thread.