Questions regarding segment-wise CLIP score computation & VBench reproduction (Table 1)

[tj12323]

Hi, thank you for releasing this excellent work!

I am currently working on reproducing the evaluation metrics reported in your paper. I have run into two areas where my reproduced numbers differ from the reported results, and I was hoping to get some clarification.

1. Segment-wise CLIP Score Computation

I am trying to reproduce the segment-wise CLIP scores (the scores computed for each prompt-aligned 10s clip, e.g., 0-10s, 10-20s, etc.). Based on the paper, my understanding of the pipeline is:

1. A 60s video is split into prompt-aligned clips based on the prompt switching boundaries.
2. Each clip is paired with its corresponding text prompt.
3. A CLIP score is computed for each clip-text pair and then reported segment-wise.

Since the exact implementation details for this metric aren't currently in the repository, my current implementation makes the following assumptions:

• Model: OpenCLIP ViT-B-32 with OpenAI pretrained weights.
• Encoding: Both text and sampled video frames are encoded with CLIP, and both features are L2-normalized.
• Sampling: Inside each segment, frames are uniformly sampled at sample_fps=2.
• Calculation: Cosine similarity is computed between each sampled frame and the corresponding prompt text. These frame-text similarities are averaged to obtain one CLIP score for the segment.
• Aggregation: Finally, the segment scores are averaged across all videos.

Here is a comparison of the scores I am getting versus the ones reported in the paper:

Model	0-10s	10-20s	20-30s	30-40s	40-50s	50-60s
MemFlow_reproduce	32.96	31.12	30.66	30.14	30.23	29.84
MemFlow (Paper)	26.31	24.70	23.94	24.13	24.90	24.22

Question: Would it be possible to share or release the evaluation script used to compute these CLIP scores? Alternatively, if you spot an obvious flaw in my assumptions above, I would greatly appreciate any clarification.

---

2. VBench Results Reproduction (Table 1)

I am also trying to reproduce the VBench results from Table 1. I generated the videos using the provided interactive_inference.sh and interactive_benchmark.jsonl, and then evaluated them using vbench2_beta_long/eval_long.py.

While my reproduced Consistency Score closely matches the paper, there is a noticeable gap in the Quality Score and Aesthetic Quality:

Metric	Quality Score	Consistency Score	Aesthetic Quality
MemFlow_reproduce	78.71	96.71	58.62
MemFlow (Paper)	85.02	96.60	61.07

For full context, here are the detailed sub-metrics from my run:

{
  'subject consistency': 0.97631974995745, 
  'background consistency': 0.9577979119413649, 
  'motion smoothness': 0.9906170158707058, 
  'dynamic degree': 0.257, 
  'aesthetic quality': 0.5862120025157929, 
  'imaging quality': 0.7226334088114897
}
# long video quality score: 0.787064055620197

Question: Could you provide some insights into what might be causing this discrepancy? Are there specific hyperparameters, seed settings, generation configs, or a specific VBench version that I should be using to perfectly align with the Table 1 results?

Having access to the exact evaluation scripts or configurations would greatly help ensure consistent comparison and reproducibility.

Thank you very much for your time and for sharing this work!