ROADMAP 260304

[chenhaot]

• Evaluation
  One of the big problem in this space is that there is no public benchmark for what thorough reviews should look like. We should have a scalable way to collect this benchmark.

• Algorithm design
  The current approach uses incremental summarization. It will have trouble for long-term dependency.
  Test recursive language modeling for this task.
  Turn the key idea into a claude skill
  Integrate with https://github.com/ChicagoHAI/MechEvalAgent/ to have execution-grounded evaluation

• Interaction
  This tool would be much more useful if the users can address the comments interactively.

• Improve comment quality
  We need a way to evaluate the quality of the overall feedback and individual comments and improve them.

[joe32140]

Turn the prompt workflow into skills in #25

[yanaiela]

Re the evaluation, I opened a PR for tracking down the comments #30
And I just thought it could be useful to add another feature (button) where a user can mark a comment as not relevant.
If you can collect such interactions, it may serve as a (silver) benchmark.
The main challenge would be to collect this data (as I guess this is currently mostly used locally?) but perhaps this could even be done with a few experts.

[dangng2004]

Re: evaluation, we have been implementing a two-pronged approach:

• Outcomes study: OpenAIReview/benchmarks/conference_study. Check if reviewers give more comments on weaker papers.
• Perturbation recall: OpenAIReview/benchmarks/perturbation. Perturb parts of a paper such as equations or claims to make it incorrect and see if AI reviewers can catch those errors. The metric is recall.