Skip to content

feat(benchmark): add Qwen-Image-Bench with Q-Judger - #2137

Merged
Yunnglin merged 2 commits into
mainfrom
feat/qwen-image-bench
Nov 19, 2026
Merged

Yunnglin merged 2 commits into
mainfrom
feat/qwen-image-bench

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Summary

  • Add qwen_image_bench for 1,000 bilingual creative prompts, using the official Q-Judger through the Native Judge workflow. Support image generation and offline image manifests.
  • Match the official 0–100 aggregation and report valid counts for the overall score and each L1/L2 metric. A failed dimension leaves independent valid dimensions usable.
  • Save one preprocessed image per source image in the output directory and reference it from all judge attempts. Generate the English and Chinese benchmark pages and AIGC index entries.

Related to #1929.

Validation

  • Verified byte-identical public JSONL contents between ModelScope and the pinned Hugging Face revision. Replayed 73,890 released judge replies for 18 generators over 1,000 prompts against the upstream scorer. All 522 model/metric score and count pairs matched, including exclusion of the one truncated reply.
  • Ran the official Q-Judger weights through the PPU vLLM service. Evaluated two released images with both Chinese and English prompts. All 16 dimension requests parsed and the upstream scorer reproduced all four image scores. Replayed those saved requests after the artifact storage change and verified byte-identical transmitted images, identical prompts, and identical scores.
  • Passed 409 targeted tests, CI Lite, make lint, make lint-imports, and the generated metadata and index checks.

Official scorer and prompts · Public dataset · Official Q-Judger

Copilot AI balanced review requested due to automatic review settings November 19, 2026 08:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions github-actions Bot added area:benchmark Benchmark adapters area:docs Documentation area:core Framework and infrastructure labels Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

EvalScope AI review

Review skipped · e84ed8c02625 · 2026-11-19 09:55:49 UTC · Actions run

The PR changes 15 files and the raw diff is 53,038 bytes (51.8 KiB). The diff limit is 49,152 bytes (48 KiB).

@github-actions github-actions Bot added needs-triage Waiting for a maintainer to triage scope and evidence ai:skipped AI review skipped with the reason recorded in its summary labels Nov 19, 2026
@Yunnglin Yunnglin removed the ai:skipped AI review skipped with the reason recorded in its summary label Nov 19, 2026
@Yunnglin
Yunnglin merged commit 795ec54 into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:benchmark Benchmark adapters area:core Framework and infrastructure area:docs Documentation needs-triage Waiting for a maintainer to triage scope and evidence

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants