Issues

• HealthBench: Step 2 of the hedging_no-uncertainty_hedges consensus criterion is inverted (affects the GPT-5.4 grader)

  openai/simple-evals#125 · Filter by author kelkalot opened on Oct 2, 2026

• [Bug / Safety False Positive] GRADER_TEMPLATE in simpleqa_eval.py triggers OpenAI Safety Filter ("Cyber Abuse" violation)

  openai/simple-evals#123 · Filter by author CHOS1N11111 opened on Sep 26, 2026

• UnboundLocalError: actual_queried_prompt_messages in mgsm_eval.py when using non-OpenAI sampler

  openai/simple-evals#122 · Filter by author nightangelflowerwin-ops opened on Sep 23, 2026

• OpenAI-compat: ChatCompletionSampler plus OPENAI_BASE_URL /v1

  openai/simple-evals#120 · Filter by author cursor opened on Aug 28, 2026

• Optional EvalPort interop for SingleEvalResult / EvalResult

  openai/simple-evals#117 · Filter by author adhabnr-ux opened on Aug 22, 2026

• Reproducibility: provenance bundle + paired statistics for fixed-item evals

  openai/simple-evals#116 · Filter by author KeilerHirsch opened on Aug 15, 2026

• MATH grading takes the first "Answer:" match (not the final answer), and check_equality interpolates candidate text raw into the grader prompt

  openai/simple-evals#115 · Filter by author AUTHENSOR opened on Aug 14, 2026

• Standard deviations for healthbench are wrong

  openai/simple-evals#109 · Filter by author za3k opened on May 6, 2026

• Batching for higher throughput?

  openai/simple-evals#105 · Filter by author szhou0202 opened on Mar 11, 2026

• Regarding the environmental data issue of the web search results of BrowseComp?

  openai/simple-evals#100 · Filter by author whfeLingYu opened on Dec 4, 2025

• Answer of a question in browsecomp may be wrong

  openai/simple-evals#99 · Filter by author flibbertigibbet-Y opened on Nov 14, 2025

• What dataset did math use?

  openai/simple-evals#98 · Filter by author yongho-chang opened on Oct 29, 2025

• What to do now that simple-evals won't be updated?

  openai/simple-evals#97 · Filter by author william-max-byte opened on Oct 11, 2025
  2

• healthbench_eval can not reproduced‌ 0.67 on the gpt-5

  openai/simple-evals#96 · Filter by author hitwangshuai opened on Sep 27, 2025
  1

• Possible Typo in DROP Benchmark Accuracy

  openai/simple-evals#90 · Filter by author IkerJansa44 opened on Jun 12, 2025

• GPQA Accuracy Mismatch on Completions vs Responses API (GPT-4o August 2024)

  openai/simple-evals#89 · Filter by author ziwenseal opened on Jun 9, 2025

• Environment setup file for Healthbench

  openai/simple-evals#88 · Filter by author drixs2050 opened on Jun 6, 2025

• Cost of a Single Evaluation Using GPT-4.1 in HealthBench

  openai/simple-evals#85 · Filter by author Rorschaaaach opened on Jun 4, 2025
  1

• Medical category of questions in the dataset

  openai/simple-evals#83 · Filter by author sid-artpark opened on May 29, 2025
  1

• Clearly wrong / tampered prompt_id with "milf"

  openai/simple-evals#79 · Filter by author junghoon-son opened on May 14, 2025
  1

• HealthBench performance data json

  openai/simple-evals#78 · Filter by author yzlnew opened on May 14, 2025
  1

• Incorrect scores in HealthBench?

  openai/simple-evals#77 · Filter by author davidgilbertson opened on May 14, 2025
  4

• Access to HealthBench Dataset?

  openai/simple-evals#76 · Filter by author alt164 opened on May 13, 2025
  2

• More up to date non-OpneAI model eval results?

  openai/simple-evals#75 · Filter by author xiaoqianWX opened on May 12, 2025

• Clarification Needed: Version of ChatGPT Used as Grader in SimpleQA

  openai/simple-evals#68 · Filter by author go-with-me000 opened on May 1, 2025