Skip to content
BranchesTags

Repository files navigation

VisualNeedle Evaluation Harness

Evaluation code for VisualNeedle, a benchmark for active visual search in high-information-density scenes. The harness supports three evaluation modes, concurrent execution, a VLM-based semantic judge, and interactive HTML reports.

Contents

run_visualneedle_eval.py      # Standard evaluation (tool-enabled or no-tool)
run_crop_black_ablation.py    # Ablation: tool images replaced with black
run_text_only_eval.py         # Baseline: no image input, text-only judge
generate_html_report.py       # HTML report generator

visualneedle_eval/            # Core package
├── runner.py                 # Per-sample evaluation pipeline
├── eval_loop.py              # Async / multiprocessing scheduler
├── visualneedle_agent.py     # Model config loading, agent construction
├── matching.py               # Answer matching and normalisation
├── judge.py                  # VLM semantic judge
├── artifacts.py              # Result structures, logging, statistics
├── image_io.py               # Image I/O and resizing
├── image_registry.py         # Image provenance tracking with affine matrices
├── bbox_geometry.py          # Coordinate transforms
├── report.py                 # HTML report rendering
├── dataset.py                # Dataset loading and sampling
├── cli.py                    # Shared model-config logic for entry scripts
├── _worker.py                # Multiprocessing worker state
├── _retry.py                 # Retry logic for incomplete runs
└── _types.py                 # EvalMode type definition

configs/models/               # YAML model configuration files
Qwen-Agent/                   # Vendored Qwen-Agent runtime

Installation

# Runtime dependencies (includes the vendored Qwen-Agent)
pip install -r requirements.txt
pip install -e .

For development and tests:

pip install -r requirements-dev.txt
make test

If you prefer not to install the package, prefix every command with:

PYTHONPATH=Qwen-Agent:. python <script>.py ...

Model Configuration

All model and judge settings are declared in a YAML config file. A starter config is provided at:

configs/models/think_with_images_neibu.yaml

The config file location is resolved in this order:

  1. --config /path/to/config.yaml (CLI flag)
  2. VISUALNEEDLE_MODEL_CONFIG=/path/to/config.yaml (environment variable)
  3. The default path compiled into visualneedle_agent.py

Config structure (minimal example)

defaults:
  active_model: my-model       # used when --model is not specified
  judge_model:  my-judge-model # used when --judge-model is not specified

models:
  my-model:
    model_type: oai            # oai | claude | qwenvl_oai | ...
    model: gpt-4o
    model_server: https://api.openai.com/v1
    api_key: ${OPENAI_API_KEY} # or api_key_env: OPENAI_API_KEY
    tools:
      image_zoom_in_tool_reason:
        bbox_order: xyxy
        bbox_scale: 1000

  my-judge-model:
    model_type: oai
    model: gpt-4o-mini
    model_server: https://api.openai.com/v1
    api_key: ${OPENAI_API_KEY}

Common config flags (shared by all three evaluation scripts):

Flag Default Description
--config FILE env / built-in YAML model config file
--model NAME defaults.active_model Model name from the YAML models section
--judge-model NAME defaults.judge_model Judge model name

Evaluation Modes

Mode 1 — Standard (run_visualneedle_eval.py)

Full VQA evaluation. The model receives the image and (optionally) calls image tools such as zoom, crop, flip, and sharpen to inspect details before answering.

# Tool-enabled (default)
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 20

# No-tool: direct VL evaluation, single turn
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --no-tools \
  --concurrency 20

Mode 2 — Crop-Black Ablation (run_crop_black_ablation.py)

Identical to standard mode, except every image returned by a tool is replaced with a same-size black image before being shown to the model. Used to measure how much the model relies on tool-output visual content rather than text reasoning.

python run_crop_black_ablation.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 20

Mode 3 — Text-Only Baseline (run_text_only_eval.py)

No image is passed to the model. The prompt tells the model it has no image access and asks for its best guess from text alone. Uses a text-only judge (no image). Establishes the prior-knowledge baseline.

python run_text_only_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 20

--no-tools and --data-format are accepted for CLI compatibility but are always ignored in text-only mode.

All CLI Flags

The flags below apply to all three evaluation scripts unless noted.

Dataset selection

Flag Default Description
--dataset-file FILE built-in default Path to a JSONL dataset
--limit N — Evaluate at most N samples (smoke test)
--start N 0 Start index (ignored when --indices is set)
--indices 0,12,99 — Evaluate specific indices only; overrides --start/--limit

Concurrency

Two concurrency modes are mutually exclusive. Use one or the other:

Flag Default Description
--concurrency N 0 (disabled) Async I/O mode — recommended for API-bound evaluation. N = max concurrent requests (20–50 is typical)
--workers N 1 Multiprocessing mode — spawns N worker processes. Use when you need process isolation

When both are set, --concurrency takes precedence.

Logging

Flag Default Description
--log-dir DIR — Directory for per-sample logs and structured results
--auto-log-dir / --no-auto-log-dir enabled Auto-generate a timestamped log directory. Disable for maximum speed
--verbose off Print full per-sample logs to stdout (very slow with concurrency > 1)
--ordered-output / --no-ordered-output auto (TTY-detect) Print results in dataset-index order when piping output to a file

Evaluation behaviour

Flag Default Description
--judge / --no-judge enabled Call VLM judge when exact/normalised match fails
--no-tools off Disable tool calling; evaluate in single-turn VL mode (standard only)
--multiple-choice off Accept option letter (A/B/C/D) as correct in addition to full answer text
--max-null-retries N 3 Max prompt variants to try when prediction is None
--data-format local|url|file local Image delivery mode (see Image Data Modes)

Retry mode (standard and ablation scripts only)

Flag Description
--retry-dir DIR Re-evaluate missing / pred=None samples from an existing log directory in-place; backs up and rebuilds summary files 
# Re-run only the failed samples from a previous experiment
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --retry-dir runs/my-model_20250518_120000

Dataset Format

The dataset must be a JSONL file with one sample per line. Required fields:

{
  "question":  "What is written on the sign in the upper-left corner?",
  "answer":    "EXIT",

  // One of the following image fields:
  "image":     "/absolute/path/to/image.jpg",    // local file path
  "image":     "data:image/jpeg;base64,/9j/...", // base64 data URI
  "image_url": "https://example.com/image.jpg"   // remote URL (--data-format url)
}

Optional fields:

{
  "answer_letter": "A",       // correct option letter for --multiple-choice
  "prompt_sn":     "q-0042",  // sample identifier shown in logs
  "question_en":   "...",     // English fallback for question
  "answer_en":     "..."      // English fallback for answer
}

Image Data Modes (--data-format)

Mode Flag Description
local --data-format local Default. Images are loaded from disk and base64-encoded into the request. Works with any provider.
url --data-format url image_url field is passed directly to the model. Tool-output images are uploaded via object_storage_upload.py — implement upload_image_to_object_storage(local_path, object_key) → url for your storage backend.
file --data-format file Claude / Anthropic Files API only. Images are uploaded once as file_id blocks and reused across retries.

Output Files

When --log-dir (or --auto-log-dir) is set, the following files are written:

<log-dir>/
├── results_summary.txt           # Human-readable per-sample summary line
├── structured_results.json       # Machine-readable full results (all fields)
├── sample_000000/
│   ├── sample_000000.log         # Full reasoning trace for this sample
│   ├── original_image.*          # Copy of the input image
│   └── (tool output images)
├── sample_000001/
│   └── ...
└── html/                         # Generated by generate_html_report.py
    ├── index.html
    ├── sample_000000.html
    └── ...

structured_results.json is written after the run completes and contains prediction, gold answer, correctness flag, tool steps with bbox coordinates, token usage, and judge results for every sample.

HTML Reports

After a run completes, generate an interactive HTML report with:

python generate_html_report.py path/to/run_logs

Optional flags:

Flag Default Description
--html-dir DIR <run-dir>/html Output directory for HTML files
--max-image-bytes N 12000000 Max bytes per inlined image (base64)
--allow-remote-images off Allow HTML to reference remote URLs from structured results
--limit N — Generate only the first N sample pages

The report is fully self-contained (all images embedded as base64) and works offline.

Typical Workflow

# 1. Smoke test — validate config and dataset loading, 5 samples
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --limit 5 --no-auto-log-dir

# 2. Full standard run with async concurrency
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 30 \
  --judge

# 3. Re-run failed samples from the same directory
python run_visualneedle_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --retry-dir runs/my-model_20250518_120000

# 4. Ablation: same setup but tool images blacked out
python run_crop_black_ablation.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 30

# 5. Text-only baseline
python run_text_only_eval.py \
  --config configs/models/think_with_images_neibu.yaml \
  --model my-model \
  --dataset-file data/visualneedle.jsonl \
  --concurrency 30

# 6. Generate HTML report
python generate_html_report.py runs/my-model_20250518_120000

Makefile Shortcuts

make install       # pip install -r requirements.txt && pip install -e .
make install-dev   # install + test dependencies
make test          # run pytest
make cli-help      # show run_visualneedle_eval.py --help
make smoke-config  # validate config loading with --limit 0
make clean         # remove __pycache__, .pytest_cache, *.egg-info

Using as a Python Library

from visualneedle_eval import evaluate_split, EvalMode, EvalResult

correct, match_correct, total, judge_used, judge_stats, tool_stats = evaluate_split(
    dataset_file="data/visualneedle.jsonl",
    limit=50,
    start=0,
    indices=None,
    verbose=False,
    concurrency=20,
    log_dir=Path("runs/my-experiment"),
    judge=True,
    mode="standard",        # "standard" | "crop_black" | "text_only"
    multiple_choice=False,
)

accuracy = correct / total * 100
print(f"Accuracy: {accuracy:.2f}%  ({correct}/{total})")
judge_stats.print_stats()
tool_stats.print_stats()

About

A benchmark for active visual search in high-information-density scenes

Resources

Readme
License
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages