MAD-RAG

This repository contains the MAD-RAG evaluation scripts for Qwen2.5-VL and LLaVA-1.5 on OKVQA, Encyclopedic-VQA, and InfoSeek.

Repository Layout

MAD-RAG/
  Qwen25vl/
    run_mad_rag.py
  llava15/
    run_mad_rag.py
  dataset/
    okvqa_test_best_of_n.jsonl
    evqa_test.jsonl
    infoseek_processed_test.jsonl
    infoseek_clip_large_retrieved1.jsonl

The dataset/*.jsonl files contain the evaluation samples and oracle chunks. The image files are not included in this repository.

Dataset Sources

The oracle chunks used by this project follow the ReT-M2KR test files:

• Oracle chunk source: https://huggingface.co/datasets/aimagelab/ReT-M2KR/tree/main/jsonl/m2kr/test

Image sources:

• OKVQA images: https://huggingface.co/datasets/BByrneLab/M2KR_Images/tree/main/OKVQA
• Encyclopedic-VQA images: https://github.com/google-research/google-research/tree/master/encyclopedic_vqa
• InfoSeek / OVEN images: https://huggingface.co/datasets/ychenNLP/oven/blob/main/all_wikipedia_images.tar

Image Directory Setup

Use any local image directory names you want, then pass them through the command-line arguments. A clean layout is:

MAD-RAG/
  images/
    okvqa/
      val2014/
        COCO_val2014_000000297147.jpg
    evqa/
      iNaturalist_2021/
      Google_Landmarks_v2/
    infoseek/
      01/
      02/
      ...

Expected root arguments:

• --okvqa_image_root should point to the directory containing val2014/.
• --evqa_image_root should point to the directory containing iNaturalist_2021/ and/or Google_Landmarks_v2/.
• --infoseek_image_root should point to the directory whose first-level subfolders contain the InfoSeek image files. The loader searches those subfolders for filenames from the real_image_path field.

By default, the scripts use these relative paths: dataset/, images/okvqa, images/evqa, images/infoseek, and output/<model_name>.

Example:

mkdir -p images/okvqa images/evqa images/infoseek

For OKVQA, download the OKVQA image zip files from Hugging Face and extract them so that val2014/ is directly under your OKVQA image root:

huggingface-cli download BByrneLab/M2KR_Images \
  --repo-type dataset \
  --include "OKVQA/val2014.zip" \
  --local-dir downloads/M2KR_Images

unzip downloads/M2KR_Images/OKVQA/val2014.zip -d images/okvqa

For EVQA, follow the official Encyclopedic-VQA instructions and organize the downloaded images so that iNaturalist_2021/ and Google_Landmarks_v2/ are under the same root:

images/evqa/iNaturalist_2021/...
images/evqa/Google_Landmarks_v2/...

For InfoSeek, download and extract the OVEN image archive. The Hugging Face dataset may require login/access approval.

huggingface-cli login
huggingface-cli download ychenNLP/oven \
  all_wikipedia_images.tar \
  --repo-type dataset \
  --local-dir downloads/oven

tar -xf downloads/oven/all_wikipedia_images.tar -C images/infoseek

If extraction creates an extra top-level folder, pass that nested folder as --infoseek_image_root.

Installation

The provided requirements.txt was generated from the original ret2 environment. The recommended conda environment name is madrag.

conda create -n madrag python=3.10 -y
conda activate madrag

pip install --upgrade pip setuptools wheel
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
  --index-url https://download.pytorch.org/whl/cu124

pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install -r requirements.txt

The requirements.txt file also includes the CUDA 12.4 PyTorch wheel index. If your machine uses a different CUDA version, install the matching PyTorch build first, then install the remaining packages from requirements.txt.

The LLaVA script uses Hugging Face LlavaForConditionalGeneration. The Qwen script uses Qwen2_5_VLForConditionalGeneration.

Running Qwen2.5-VL

OKVQA:

python Qwen25vl/run_mad_rag.py \
  --dataset okvqa \
  --model_path Qwen/Qwen2.5-VL-7B-Instruct \
  --okvqa_image_root images/okvqa \
  --output_dir output/Qwen25vl

EVQA:

python Qwen25vl/run_mad_rag.py \
  --dataset evqa \
  --model_path Qwen/Qwen2.5-VL-7B-Instruct \
  --evqa_image_root images/evqa \
  --output_dir output/Qwen25vl

InfoSeek:

python Qwen25vl/run_mad_rag.py \
  --dataset infoseek \
  --model_path Qwen/Qwen2.5-VL-7B-Instruct \
  --infoseek_image_root images/infoseek \
  --output_dir output/Qwen25vl

Running LLaVA-1.5

OKVQA:

python llava15/run_mad_rag.py \
  --dataset okvqa \
  --model_path llava-hf/llava-1.5-7b-hf \
  --okvqa_image_root images/okvqa \
  --output_dir output/llava15

EVQA:

python llava15/run_mad_rag.py \
  --dataset evqa \
  --model_path llava-hf/llava-1.5-7b-hf \
  --evqa_image_root images/evqa \
  --output_dir output/llava15

InfoSeek:

python llava15/run_mad_rag.py \
  --dataset infoseek \
  --model_path llava-hf/llava-1.5-7b-hf \
  --infoseek_image_root images/infoseek \
  --output_dir output/llava15

Useful Arguments

• --dataset: one of okvqa, evqa, infoseek
• --dataset_dir: directory containing the JSONL files, defaulting to dataset/
• --okvqa_file_path, --evqa_file_path, --infoseek_file_path: override individual JSONL files
• --okvqa_image_root, --evqa_image_root, --infoseek_image_root: local image roots
• --use_rag: enable RAG context
• --enable_injection: enable attention injection
• --injection_alpha: injection strength
• --top_k_chunks: number of oracle chunks to use
• --start, --end: evaluate a subset
• --output_dir: where result JSON files are written

Notes

The scripts do not download model weights or images automatically. Keep image files outside Git and pass their local paths with the image root arguments. Generated outputs are ignored by .gitignore.