feat(e2e-eval): support pre-exported ONNX models via onnx_file#902
Open
xieofxie wants to merge 3 commits into
Open
feat(e2e-eval): support pre-exported ONNX models via onnx_file#902xieofxie wants to merge 3 commits into
xieofxie wants to merge 3 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 2 commits
June 16, 2026 16:28
CodeQL flagged 'onnx' being imported with both 'import onnx' and 'from onnx import ...' in the same module. Use attribute access (onnx.version_converter / onnx.helper / onnx.TensorProto) throughout instead of mixing import styles.
The Phase 2 onnx_file passthrough was dead: load_curated_entries dropped every key except hf_id/task/group/priority, so a rebuild that re-added or newly-created a curated onnx_file entry would lose it. Carry onnx_file through when present.
Contributor
Author
|
need to update this in favor of #582 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for evaluating models that ship a pre-exported ONNX (e.g. the PaddleOCR
*_onnxrepos) in the e2e_eval harness, bypassing the HF→ONNX export path entirely. Motivated byPaddlePaddle/PP-OCRv5_server_{det,rec}, whose nativetransformersarchitecture is unsupported by the pinnedtransformers/optimum-onnxstack (a hard dependency conflict: the model needstransformers>=5, whileoptimum-onnxcaps it at<4.58).When a registry entry declares
onnx_file, the harness downloads that file from the HF repo and feeds the local path towinml config/build/perfvia-m(winml'sis_onnx_file_pathroutes it to the skip-export pipeline).Changes
models_curated.json,models_all.json): renamedPP-OCRv5_server_{det,rec}→*_onnxwith"onnx_file": "inference.onnx".utils/registry.py: newModelEntry.onnx_filefield + load mapping.build_registry.py: flowsonnx_filethrough Phase 2 (update-in-place + new-entry) so it survives rebuilds.run_eval.py:_resolve_model_input()downloads the pre-exported ONNX and returns the local path (else hf_id); used for bothconfig -mandbuild -m. Prepare failures degrade to a graceful per-model build failure instead of crashing the run._ensure_min_opset()upgrades sub-minimum ONNX to opset 17 (winml optimize requires >=12; PaddleOCR ships opset 11). onnxruntime can run opset 11, but the build/optimize stage rejects it.--output-dirinstead of--use-cache(direct-ONNX configs carry noloader.taskto key the cache), and resolve the artifact deterministically at<output-dir>/model.onnx(stdout marker parsing was unreliable — Rich wraps long paths, which silently dropped perf to a build-only false PASS).--hf-model+--tasknow usesdataclasses.replacesoonnx_file/precision/perf_argssurvive the task override.Verification
run_eval.py --hf-model PaddlePaddle/PP-OCRv5_server_det_onnx --eval-type perf --device cpu→ PASS, real perf on the builtmodel.onnx(5.40 samples/sec, full latency table).tests/unit/eval/test_run_eval_script.py: 23 passed (added coverage for opset-upgrade, download-then-ensure-opset, and the--output-dir/-mwiring).