Questions about the generation and meaning of multiple txt files (e.g. relative_test3_NCEV.txt etc.)

[muppychae1]

Hello, author.
I am trying to reimplement Emotion-LLaMA and test its performance on other datasets, but I had a few questions while writing the code.

When doing evaluation, I saw files like (MER2023)relative_test3_NCEV.txt, MERR_fine_grained.txt, (MER2024)MER2024-noise_NCE.txt, and (DFEW)reltive_DFEW_set_1_test.txt, but I am not quite sure how they are structured.

For example, in part of the content of MERR_fine_grained.txt, I found entries like the following. How are the columns organized in this case?

sample_00000023 35 angry -1.174107
sample_00000046 98 angry -2.5923429
sample_00000047 22 surprise -0.61232674
sample_00000054 52 angry -1.0886253
sample_00000069 18 angry -1.5699985

And how can i make those txt files like that?

I look forward to your reply. Thank you very much!

[ZebangCheng]

Hello, and thank you for your interest in our work.

Regarding the annotation files, we followed the convention from previous studies, where the “NCEV” format includes four columns:

• N: the video sample name
• C: the number of frames in the video
• E: the emotion category label (or pseudo-label) of the video
• V: the valence value (or pseudo-label) of the video’s emotion

However, in the Emotion-LLaMA project, we only use N and E. Therefore, when you reimplement Emotion-LLaMA on new datasets, you only need to construct annotation files containing the video sample name and the corresponding emotion category label.

[muppychae1]

Thank you very much for your detailed answer!

I have one more question.
You have kindly shared the feature extraction code on Google Drive, so I was able to extract features using MAE and VideoMAE. However, I haven’t been able to generate the part called “HL_UTT” yet. Could you let me know which model should be used to extract this part of the features and how to do this process?

[liu-yuyun]

Hello, I couldn't find "relative_test3_NCEV.txt" in the project. Where can I get this file?

[ZebangCheng]

Thank you very much for your detailed answer!

I have one more question. You have kindly shared the feature extraction code on Google Drive, so I was able to extract features using MAE and VideoMAE. However, I haven’t been able to generate the part called “HL_UTT” yet. Could you let me know which model should be used to extract this part of the features and how to do this process?

---

To extract the “HL_UTT” features, please refer to the MER2023 baseline code:
https://github.com/zeroQiaoba/MERTools/blob/master/MER2023/feature_extraction/audio/extract_transformers_embedding.py

[ZebangCheng]

Hello, I couldn't find "relative_test3_NCEV.txt" in the project. Where can I get this file?

---

"relative_test3_NCEV.txt" link:

https://drive.google.com/file/d/1YyoWabWtAJuFI6ylMM220i_kh_0Nagv9/view?usp=sharing

[muppychae1]

Then, if I want to evaluate performance on a new dataset, would it be better to use the model trained on MER2023 (checkpoint_best.pth) or the one trained on MER2024 (MER2024-best.pth)?

Based on new dataset, when I evaluated using checkpoint_best.pth, I got an F1 score of 0.209, and when I used MER2024-best.pth, I got an F1 score of 0.422. Could you suggest why the MER2024 model performs a bit better than the 2023 one? (I used a command : torchrun --nproc_per_node 1 eval_emotion.py --cfg-path eval_configs/eval_emotion.yaml --dataset mer2024_caption)

Also, both F1 scores are lower than I expected—what could be the possible reasons for this? My guess is that in my dataset, each video clip contains the utterance of a single speaker, but there are often multiple faces appearing in the scene at the same time. Could this be affecting the performance?

Thank you very much for taking the time to help me:)

[ZebangCheng]

First, I recommend using MER2024-best.pth for multimodal emotion recognition. The reason is that when training MER2023 (checkpoint_best.pth), we forgot to fix the random seed during feature extraction. This caused a mismatch, as the features extracted later during inference could not be guaranteed to be identical, making it difficult to reproduce the optimal performance. In contrast, for MER2024 we fixed the random seed when extracting multimodal features, and we also released the corresponding feature extraction code on Google Drive. Therefore, the features you are currently extracting following the tutorial are consistent with those used to train MER2024-best.pth.

Second, regarding the relatively low performance of both models, the most direct reason is the distribution gap between datasets—zero-shot performance often drops significantly on new data. Here are a few suggestions:

1. Cross-model evaluation: Try evaluating other emotion recognition models (e.g., AffectGPT, R1-Omni) or general-purpose models (e.g., Qwen-2.5-Omni, GPT-4o) to establish a baseline for comparison.
2. Check input modalities: Make sure that subtitles (textual information) are provided when testing Emotion-LLaMA, as they are an important part of the model’s input.
3. Fine-tuning: Extract features from your training set and fine-tune the model for better adaptation.

I’m also very interested in learning more about the new dataset you are applying this to. Could you share what your task looks like, and how many emotion categories are included? Please feel free to reach out to me via email (zebang.cheng@gmail.com)—I’d be glad to discuss further. I truly hope Emotion-LLaMA can be applied to a wide range of domains.

[muppychae1]

Hello, author!
I sincerely apologize for checking your reply late.

I am simply aiming to perform the emotion recognition task, and as you suggested, I would like to try fine-tuning Emotion-LLaMA. Could you please let me know how to fine-tune it on a new dataset? Is it right to train Stage1 and then use train_configs/Emotion-LLaMA_finetune.yaml?

I truly appreciate your help.

[ZebangCheng]

Yes, you can start by using train_configs/Emotion-LLaMA_finetune.yaml to train Stage1. After that, you should construct an instruction dataset based on your new dataset and continue training in the same way.

[muppychae1]

So you mean that I need to follow the steps below?

1. Build a new dataset as an instruction dataset using MER-Factory.
2. Train Stage 1 based on MERR_coarse_grained.json with train_configs/Emotion-LLaMA_finetune.yaml.
3. Using the generated checkpoint_best.pth(You already give us Emotion_LLaMA.pth through GoogleDrive, right?), perform Instruction Tuning on the newly constructed instruction dataset.

• When running Stage 2 (Instruction Tuning), do I need to change the self.task_pool in first_face.py to include only emotion? The reason is that I want to focus on Emotion Recognition.