Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions docs/en/benchmarks/qwen_image_bench.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# Qwen-Image-Bench


## Overview

Qwen-Image-Bench measures how well text-to-image models handle creative tasks.

## Task Description

- **Task Type**: Text-to-image generation
- **Input**: A Chinese or English prompt
- **Output**: A generated image
- **Domain**: Image quality, aesthetics, prompt alignment, real-world fidelity, and creativity

## Key Features

- 1,000 bilingual prompts. Set `extra_params.language` to `cn` (default) or `en`.
- Score existing images with a local JSONL (`ID`, `prompt`, `dims_en`, and absolute `image_path`).
Set `model_task='image_generation'` and omit `model`.

## Evaluation Notes

- Configure `judge.models` with an OpenAI-compatible endpoint serving the official Q-Judger
(`Qwen/Qwen-Image-Bench`). See the [official inference settings](https://github.com/QwenLM/Qwen-Image-Bench#inference-parameters).
- Scores range from 0 to 100. Higher is better. The report shows overall and breakdown scores with their valid image counts.


## Properties

| Property | Value |
|----------|-------|
| **Benchmark Name** | `qwen_image_bench` |
| **Dataset ID** | [Qwen/Qwen-Image-Bench](https://modelscope.cn/datasets/Qwen/Qwen-Image-Bench/summary) |
| **Paper** | [Paper](https://arxiv.org/abs/2605.28091) |
| **Tags** | `TextToImage` |
| **Metrics** | `qwen_image_bench_score` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |


## Data Statistics

| Metric | Value |
|--------|-------|
| Total Samples | 1,000 |
| Prompt Length (Mean) | 83.05 chars |
| Prompt Length (Min/Max) | 25 / 317 chars |

## Sample Example

**Subset**: `default`

```json
{
"input": [
{
"id": "58f90ec0",
"content": "一只拆开的机械手表放在黑色工作台上,制表师用镊子夹起齿轮,特写镊子夹起齿轮,银色与暗灰色调,零部件结构精密,金属边缘细节清晰。"
}
],
"target": "",
"id": 0,
"group_id": 0,
"metadata": {
"id": 1,
"prompt": "一只拆开的机械手表放在黑色工作台上,制表师用镊子夹起齿轮,特写镊子夹起齿轮,银色与暗灰色调,零部件结构精密,金属边缘细节清晰。",
"dimensions": [
"Quality",
"Aesthetics",
"Alignment"
],
"image_path": "",
"category": "",
"tags": []
}
}
```

## Prompt Template

*No prompt template defined.*

## Extra Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `language` | `str` | `cn` | Language of the generation prompts (cn or en). Choices: ['cn', 'en'] |

## Usage

### Using CLI

```bash
evalscope eval \
--model YOUR_MODEL \
--model-task image_generation \
--eval-type text2image \
--api-url http://127.0.0.1:18131/v1 \
--api-key EMPTY_TOKEN \
--datasets qwen_image_bench \
--judge '{"strategy":"llm","models":[{"model_id":"YOUR_JUDGE_MODEL"}]}' \
--limit 10 # Remove this line for formal evaluation
```

### Using Python

```python
from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
model='YOUR_MODEL',
eval_type='text2image',
model_task='image_generation',
api_url='http://127.0.0.1:18131/v1',
api_key='EMPTY_TOKEN',
datasets=['qwen_image_bench'],
judge={'strategy': 'llm', 'models': [{'model_id': 'YOUR_JUDGE_MODEL'}]},
dataset_args={
'qwen_image_bench': {
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)
```
2 changes: 2 additions & 0 deletions docs/en/get_started/supported_dataset/aigc.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Below is the list of supported AIGC benchmarks. Click on a benchmark name for de
| `general_t2i` | [General-T2I](../../benchmarks/general_t2i.md) | `Custom`, `TextToImage` |
| `hpdv2` | [HPD-v2](../../benchmarks/hpdv2.md) | `TextToImage` |
| `longtext_bench` | [LongText-Bench](../../benchmarks/longtext_bench.md) | `TextToImage` |
| `qwen_image_bench` | [Qwen-Image-Bench](../../benchmarks/qwen_image_bench.md) | `TextToImage` |
| `tifa160` | [TIFA-160](../../benchmarks/tifa160.md) | `TextToImage` |

:::{toctree}
Expand All @@ -24,5 +25,6 @@ Below is the list of supported AIGC benchmarks. Click on a benchmark name for de
../../benchmarks/general_t2i.md
../../benchmarks/hpdv2.md
../../benchmarks/longtext_bench.md
../../benchmarks/qwen_image_bench.md
../../benchmarks/tifa160.md
:::
127 changes: 127 additions & 0 deletions docs/zh/benchmarks/qwen_image_bench.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Qwen-Image-Bench


## 概述

Qwen-Image-Bench 用于衡量文本到图像模型在处理创意任务时的表现。

## 任务描述

- **任务类型**:文本到图像生成
- **输入**:中文或英文提示词
- **输出**:生成的图像
- **领域**:图像质量、美学、提示词对齐度、真实世界保真度和创造力

## 主要特点

- 包含 1,000 条双语提示词。将 `extra_params.language` 设置为 `cn`(默认)或 `en`。
- 使用本地 JSONL 文件对现有图像进行评分(需包含 `ID`、`prompt`、`dims_en` 以及绝对路径的 `image_path`)。
设置 `model_task='image_generation'` 并省略 `model` 参数。

## 评估说明

- 配置 `judge.models`,指向提供官方 Q-Judger (`Qwen/Qwen-Image-Bench`) 服务的 OpenAI 兼容端点。请参阅[官方推理设置](https://github.com/QwenLM/Qwen-Image-Bench#inference-parameters)。
- 分数范围为 0 到 100,分数越高越好。报告将显示总体得分及细分得分,并列出有效图像的数量。


## 属性

| 属性 | 值 |
|----------|-------|
| **基准测试名称** | `qwen_image_bench` |
| **数据集ID** | [Qwen/Qwen-Image-Bench](https://modelscope.cn/datasets/Qwen/Qwen-Image-Bench/summary) |
| **论文** | [Paper](https://arxiv.org/abs/2605.28091) |
| **标签** | `TextToImage` |
| **指标** | `qwen_image_bench_score` |
| **默认示例数** | 0-shot |
| **评估划分** | `test` |


## 数据统计

| 指标 | 值 |
|--------|-------|
| 总样本数 | 1,000 |
| 提示词长度(平均值) | 83.05 字符 |
| 提示词长度(最小/最大值) | 25 / 317 字符 |

## 样例示例

**子集**: `default`

```json
{
"input": [
{
"id": "58f90ec0",
"content": "一只拆开的机械手表放在黑色工作台上,制表师用镊子夹起齿轮,特写镊子夹起齿轮,银色与暗灰色调,零部件结构精密,金属边缘细节清晰。"
}
],
"target": "",
"id": 0,
"group_id": 0,
"metadata": {
"id": 1,
"prompt": "一只拆开的机械手表放在黑色工作台上,制表师用镊子夹起齿轮,特写镊子夹起齿轮,银色与暗灰色调,零部件结构精密,金属边缘细节清晰。",
"dimensions": [
"Quality",
"Aesthetics",
"Alignment"
],
"image_path": "",
"category": "",
"tags": []
}
}
```

## 提示模板

*未定义提示模板。*

## 额外参数

| 参数 | 类型 | 默认值 | 描述 |
|-----------|------|---------|-------------|
| `language` | `str` | `cn` | 生成提示词的语言(cn 或 en)。可选值:['cn', 'en'] |

## 使用方法

### 使用 CLI

```bash
evalscope eval \
--model YOUR_MODEL \
--model-task image_generation \
--eval-type text2image \
--api-url http://127.0.0.1:18131/v1 \
--api-key EMPTY_TOKEN \
--datasets qwen_image_bench \
--judge '{"strategy":"llm","models":[{"model_id":"YOUR_JUDGE_MODEL"}]}' \
--limit 10 # 正式评估时请删除此行
```

### 使用 Python

```python
from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
model='YOUR_MODEL',
eval_type='text2image',
model_task='image_generation',
api_url='http://127.0.0.1:18131/v1',
api_key='EMPTY_TOKEN',
datasets=['qwen_image_bench'],
judge={'strategy': 'llm', 'models': [{'model_id': 'YOUR_JUDGE_MODEL'}]},
dataset_args={
'qwen_image_bench': {
# extra_params: {} # uses default extra parameters
}
},
limit=10, # 正式评估时请删除此行
)

run_task(task_cfg=task_cfg)
```
2 changes: 2 additions & 0 deletions docs/zh/get_started/supported_dataset/aigc.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
| `general_t2i` | [General-T2I](../../benchmarks/general_t2i.md) | `Custom`, `TextToImage` |
| `hpdv2` | [HPD-v2](../../benchmarks/hpdv2.md) | `TextToImage` |
| `longtext_bench` | [LongText-Bench](../../benchmarks/longtext_bench.md) | `TextToImage` |
| `qwen_image_bench` | [Qwen-Image-Bench](../../benchmarks/qwen_image_bench.md) | `TextToImage` |
| `tifa160` | [TIFA-160](../../benchmarks/tifa160.md) | `TextToImage` |

:::{toctree}
Expand All @@ -24,5 +25,6 @@
../../benchmarks/general_t2i.md
../../benchmarks/hpdv2.md
../../benchmarks/longtext_bench.md
../../benchmarks/qwen_image_bench.md
../../benchmarks/tifa160.md
:::
4 changes: 4 additions & 0 deletions evalscope/api/judge/contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ class OutputContract(BaseModel):
model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)

schema_model: Type[BaseModel]
allow_json_fences: bool = False

def instruction(self) -> str:
"""The format requirement appended to the benchmark's own prompt."""
Expand All @@ -56,6 +57,9 @@ def parse(self, response: str) -> ParseResult:
if not response or not response.strip():
return ParseResult.failure('empty response')

if self.allow_json_fences:
response = response.strip().removeprefix('```json\n').removeprefix('```\n').removesuffix('```').strip()

try:
data = json.loads(
response,
Expand Down
4 changes: 3 additions & 1 deletion evalscope/api/judge/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ def _run_observation(
for case in pending:
verdict = self._run_case(definition, context, judge, repeat_id, case, completed, review)
if verdict is None:
if not case.required:
continue
observation.status = ScoreStatus.INVALID_SESSION
observation.error = f'case {case.case_id} produced no usable verdict'
return observation
Expand Down Expand Up @@ -274,7 +276,7 @@ def _aggregate(self, review: JudgeReview) -> None:
if tie_broken:
review.metadata['tie_broken_by_primary'] = True
review.status = ScoreStatus.DEGRADED
elif len(review.valid_observations) != expected or unresolved_ties or repeat_ties:
elif len(review.valid_observations) != expected or review.failure_counts or unresolved_ties or repeat_ties:
review.status = ScoreStatus.DEGRADED

def _aggregate_rule_fallback(self, review: JudgeReview) -> None:
Expand Down
3 changes: 3 additions & 0 deletions evalscope/api/judge/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ class JudgeCase(BaseModel):
output_contract: OutputContract
"""Declares the verdict shape and is the only thing allowed to parse the response."""

required: bool = True
"""Allow other independent cases to contribute if this case fails."""

metadata: Dict[str, Any] = Field(default_factory=dict)
"""Adapter-defined data carried through the request callback and onto the verdict."""

Expand Down
1 change: 1 addition & 0 deletions evalscope/benchmarks/_index.json
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@
"process_bench": "evalscope.benchmarks.process_bench.process_bench_adapter",
"pubmedqa": "evalscope.benchmarks.pumed_qa.pubmed_qa_adapter",
"qasc": "evalscope.benchmarks.qasc.qasc_adapter",
"qwen_image_bench": "evalscope.benchmarks.qwen_image_bench.qwen_image_bench_adapter",
"race": "evalscope.benchmarks.race.race_adapter",
"real_world_qa": "evalscope.benchmarks.real_world_qa.real_world_qa_adapter",
"ref_adv_s": "evalscope.benchmarks.ref_adv_s.ref_adv_s_adapter",
Expand Down
Loading
Loading