[pull] main from open-compass:main by pull[bot] · Pull Request #25 · changlan/opencompass

pull · 2025-05-22T13:04:18Z

See Commits and Changes for more details.

Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

* update * fix lint * fix lint * update precommit * update precommit * fix lint

* 0-shot Smolinstruct Add 0-shot evaluation and postprocess functions for Smolinstruct * fix acc postprocessor * update 0-shot acc postprocessor * rename 0-shot

* 250527 * 250527 * 250527 * 0530 * 0530 * Update srbench_gen.py * Update srbench.py * srbench fix * Update datasets_info.py * Update datasets_info.py --------- Co-authored-by: Myhs-phz <demarcia2014@126.com> Co-authored-by: Linchen Xiao <xxllcc1993@gmail.com>

* add LLM as judge setting for matbench * fix reference negative sample gold value missing error * update import * comments trim * fix file and import naming * matbench fix * matbench fix * matbench fix --------- Co-authored-by: Jucheng Hu <jucheng.hu.20@ucl.ac.uk> Co-authored-by: Myhs-phz <demarcia2014@126.com>

* add phybench * phybench fix * update * update --------- Co-authored-by: Myhs-phz <demarcia2014@126.com> Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

* update * update * update

* add gaokao & competition benchmark * fix lint * chem_exam fix --------- Co-authored-by: Myhs-phz <demarcia2014@126.com>

* add gaokao & competition benchmark * fix lint * chem_exam fix * update verifier prompt --------- Co-authored-by: Myhs-phz <demarcia2014@126.com>

* update earth silver benchmark * fix new issues * update * update --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

* healthbench * fix irrelevant files * first * fix bench * fix bench * fix bench * fix soft link * fix bench * fix bench * healthbench fix * fix bench * fix bench * fix bench * fix bench * fix bench * fix bench * fix bench * fix bench * fix bench --------- Co-authored-by: Myhs-phz <demarcia2014@126.com>

* update * update * update * update * update

* [Dataset] Add R-Bench (ICML 2025) * fixed lint * format rbench.py by isort * rbench fix * r-bench fix * update --------- Co-authored-by: leoyizhang <leoyizhang@tencent.com> Co-authored-by: Myhs-phz <demarcia2014@126.com> Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

* Fix PHYbench * update --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

* 0616 * 0616 * 0616 * update * update * 0616 --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

* update * update

* update needlebench docs for chinese * update bilingual needlebench docs * update docs typo * update docs * update docs typo * [Docs] fix needlebench examples * Add NeedleBench_V2 * [Fix] Fix pre-commit * remove choice version * [Docs] Update NeedleBench Docs * [Docs] update NeedleBenchV2 Docs * [Docs] Update Default Settings for NeedleBench and ATC Configs * [Fix] Fix precommit * [Minor] fix needlebench summarizer groups * [Minor] Update NeedleBenchV2 dataset-index

* timed re.search and _executor made global * TimeOutError exception handling * added missing blank lines * isort import --------- Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>

* update * update

* debug rjob runner * optimize concurrent requests by adding max_workers * update --------- Co-authored-by: xujun <xujun@pjlab.org.cn>

* debug rjob runner * optimize concurrent requests by adding max_workers * update * optimize the max_workers for OpenAISDK * optimize the max_workers for OpenAISDK * optimize the max_workers for OpenAISDK * optimize the max_workers for OpenAISDK * Update openai_api.py --------- Co-authored-by: xujun <xujun@pjlab.org.cn> Co-authored-by: nic <nic@yccc.follower> Co-authored-by: zhulinJulia24 <145004780+zhulinJulia24@users.noreply.github.com>

(Warning) This PR may introduce performance BC for ChemExam benchmark.

… LCBV6 (#2393) * fix * fix * fix lint

* add example * fix

* Fix extract_role_pred to properly strip whitespace and use explicit None checks * update * fix lint * update * update * update * update * update * update * update * update * update * Update test_base_task.py * Update test_base_task.py

* add models unittest * add more unit testcase * refactor ete testcase * update * update * fix unfail import * update mount * update url * fix import * update * update * Revert to relative imports with __init__.py support The __init__.py files in autotest and autotest/infer packages enable relative imports to work correctly even in mmengine's lazy loading context. This is more robust than absolute imports which can fail when sys.path doesn't include the opencompass root. * updaste * Add proper __init__.py files to prevent namespace package issues Add docstrings and __all__ to make autotest a proper package instead of namespace package * add init * Fix TypeError: LazyObject is not iterable Remove filtering of datasets at module level as it tries to iterate LazyObject before it's built by mmengine. Simply assign datasets = datasets to preserve the lazy loading behavior. * fix import * update * update * fix eval path * fix compare * update import * update * update * update * update * update configs * update * update config * update * update * update * update * upda * fix * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update path * update

* add * fix * fix * add * fix * fix * add unitest * delete * add * fix * add all dataset config

* update * update

…ChatMLDatasets (#2414) * fix * fix * fix

* [Refactor] Support concurrent inference accorss tasks. * Rename inferencer and fix openai streaming sdk * fix lint * [Fix] Add parallel ChatML inferencer and JSONL checkpoints * fix lint * Fix unit tests * add more ut --------- Co-authored-by: zhulin1 <zhulinJulia24@163.com>

* update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update

* add molecular_iq evaluation * fix molecular_core * fix molecular_core * update * Delete opencompass/datasets/moleculariq/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/_data/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/_dynamic/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/_nlp/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/_pools/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/rewards/__pycache__ directory * Delete opencompass/datasets/moleculariq/moleculariq_core/solver/__pycache__ directory * Add error handling for RDKit imports Wrap RDKit imports in a try-except block to handle potential import errors. * Add error handling for RDKit import Handle import error for RDKit library gracefully. * Update template_based_reaction_solver.py * Refactor SymbolicSolver class and bond patterns * Refactor RDKit import exception handling Moved exception handling for RDKit imports to the end and added initialization for SymbolicSolver class. * Add error handling for RDKit imports Handle import errors for RDKit library gracefully. --------- Co-authored-by: Myhs_phz <demarcia2014@126.com>

…romptTemplate (#2422) * add * fix lint

* fix * fix unit test * fix lint * fix lint

[ci] update dlc setting (#2112)

c3779eb

pull Bot added the ⤵️ pull label May 22, 2025

pull Bot had a problem deploying to prod May 22, 2025 13:04 Error

Myhs-phz and others added 27 commits May 27, 2025 19:41

add qwen3 lmdeply (#2126)

6f3c670

[Dataset] Add SuperGPQA subfield configs (#2124)

408f5ca

* update * fix lint * fix lint * update precommit * update precommit * fix lint

[Dataset] Add Smolinstruct configs (#2127)

d572761

* 0-shot Smolinstruct Add 0-shot evaluation and postprocess functions for Smolinstruct * fix acc postprocessor * update 0-shot acc postprocessor * rename 0-shot

[Datasets] Add PHYBench (#2125)

80ec846

* add phybench * phybench fix * update * update --------- Co-authored-by: Myhs-phz <demarcia2014@126.com> Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

[Update] Minor Updates (#2136)

5720ebf

* update * update * update

[Fix] Delete wrong internvl config (#2135)

8379c4b

[Feature] Add Chem Gaokao & Competition Benchmark (#2134)

1b34aaf

* add gaokao & competition benchmark * fix lint * chem_exam fix --------- Co-authored-by: Myhs-phz <demarcia2014@126.com>

[Feature] Adjust Chem exam verifier (#2142)

da23fd9

* add gaokao & competition benchmark * fix lint * chem_exam fix * update verifier prompt --------- Co-authored-by: Myhs-phz <demarcia2014@126.com>

[Dataset] Earth Silver Benchmark (#2140)

eae3142

* update earth silver benchmark * fix new issues * update * update --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

[Update] Update Aime2024 auto-download (#2137)

c29c258

[Feature] Rjob Runner (#2144)

becff4c

* update * update * update * update * update

[Dataset] Update PHYbench postprocess (#2150)

2a1e6e8

* Fix PHYbench * update --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

[Dataset] Update Srbench dataset (#2154)

5d75dc2

* 0616 * 0616 * 0616 * update * update * 0616 --------- Co-authored-by: MaiziXiao <xxllcc1993@gmail.com>

[Update] Update SmolInsttruct dataset config (#2157)

978fc36

* update * update

[Update] Handle SRbench non-parseable pred (#2158)

944e90d

* update * update

[Dataset] Extend 256k and 512k data for RULER (#2109)

5fd4899

More stable MBPP evaluation (#2111)

d07bb3d

* timed re.search and _executor made global * TimeOutError exception handling * added missing blank lines * isort import --------- Co-authored-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>

[Dataset] Fix SuperGPQA postprocess (#2165)

a9fc798

* update * update

fix statis.py (#2170)

87ae8db

[Fix] RJob Runner and max_worker for OpenAISDK (#2171)

bc0480b

* debug rjob runner * optimize concurrent requests by adding max_workers * update --------- Co-authored-by: xujun <xujun@pjlab.org.cn>

[Fix] Fix the systemprompt issue for chem_exam benchmark(#2169)

ea02ee9

(Warning) This PR may introduce performance BC for ChemExam benchmark.

Myhs-phz and others added 30 commits February 5, 2026 18:17

[Update] Add evaluation example for Intern-S1-Pro (#2394)

bb15146

[Fix] Add custom mock for sys.stdin that supports buffer attribute in…

a292806

… LCBV6 (#2393) * fix * fix * fix lint

[Update] Add evaluation example for SciReasoner (#2395)

480c6ca

* add example * fix

[ci] add unittest (#2390)

0d99e7c

* Fix extract_role_pred to properly strip whitespace and use explicit None checks * update * fix lint * update * update * update * update * update * update * update * update * update * Update test_base_task.py * Update test_base_task.py

[Update] add LLM-judge-based C-Eval config

714b380

[Update] add finish_reason_confirm tag in OpenAISDKStreaming

2c6fb22

[Update] update subjective config in compassbench

98e29a2

[Update] add oss download for new added datasets.

d83a68d

[Bump] Bump version to 0.5.2 (#2402)

9741792

[Dataset] add AIME2026 and HMMT Feb 2026 (#2404)

cd0c6d7

[Fix] fix path of IFEval (#2406)

7a6dcf3

[Feautre] Add RawPromptTemplate and new dataset configs (#2407)

e39144f

* add * fix * fix * add * fix * fix * add unitest * delete * add * fix * add all dataset config

[Update] Reduce RJOB sleep time (#2411)

f0fd56a

* update * update

[Fix] Fix the tag match in generic_llmjudge_postprocess (#2417)

3d68e28

[Update] Add support for RawPromptTemplate in OpenAISDKStreaming and …

1799dcf

…ChatMLDatasets (#2414) * fix * fix * fix

[Fix] fix generic.py (#2419)

e12c89a

[Fix] fix chatml and dataset utils (#2421)

6a22f01

[Doc] Add guide for RawPromptTemplate (#2420)

ad7fa7e

[Fix] fix OlympiadBench with RawPromptTemplate (#2423)

9b9acfc

[Update] Add repeat config for HMMT2025 and UGD_hard (#2425)

fcbb26e

[Fix] fix ellipsis extraction in bio_data (#2429)

048a09f

[ci] add infer test case by using mock api (#2428)

c7c022a

* update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update * update

[Fix] fix moleculariq prtest (#2432)

4264cad

[Update] Add SciReasoner config and support for Retrievers using RawP…

56ee997

…romptTemplate (#2422) * add * fix lint

[Dataset] add simpleqa-verified (#2436)

38b23bf

[Fix] add multi round support from api_meta_template (#2435)

8add477

* fix * fix unit test * fix lint * fix lint

[Fix] fix result metrics in SimpleQA (#2439)

0524d49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pull] main from open-compass:main#25

[pull] main from open-compass:main#25
pull[bot] wants to merge 160 commits intochanglan:mainfrom
open-compass:main

pull Bot commented May 22, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

pull Bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

pull Bot commented May 22, 2025 •

edited

Loading