KeyError: 'scale_invariant_signal_noise_ratio' in test_epoch function

[YuanxinGuo]

I'm encountering an issue while training the Waveformer model. When I run the following command:

python -W ignore -m src.training.train /home/swufe1/project/Waveformer/experiments/dcc_tf_ckpt_E256_10_D256_1 --use_cuda

I receive the following error message:

2024-10-31 10:27:40.500632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
Imported the model from 'src.training.dcc_tf'.
Loading train dataset: fg_dir=data/FSDSoundScapes/FSDKaggle2018/train bg_dir=data/FSDSoundScapes/TAU-acoustic-sounds/TAU-urban-acoustic-scenes-2019-development
Loaded train dataset at data/FSDSoundScapes containing 50000 elements
Loading val dataset: fg_dir=data/FSDSoundScapes/FSDKaggle2018/val bg_dir=data/FSDSoundScapes/TAU-acoustic-sounds/TAU-urban-acoustic-scenes-2019-development
Loaded test dataset at data/FSDSoundScapes containing 5000 elements
Using CUDA devices: [0, 1, 2, 3]
Using data parallel model
Initializing optimizer with {'lr': 0.0005, 'weight_decay': 0.0}
Learning rates initialized to: {group 0: params=1.61230M lr=5.0E-04}
Initialized LR scheduler with params: fix_lr_epochs=50 {'mode': 'max', 'factor': 0.1, 'patience': 5, 'min_lr': 5e-06, 'threshold': 0.1, 'threshold_mode': 'abs'}
Epoch 0:
Train: 100%|███████████████████████████████████| 3125/3125 [1:19:01<00:00, 1.52s/it, loss=-0.93717]
Train: _signal_noise_ratio=7.7241 _scale_invariant_signal_noise_ratio=2.0061 loss=-0.7130
Test: 0%| | 0/79 [00:49<?, ?it/s]
Traceback (most recent call last):
File "/home/swufe1/project/Waveformer/src/training/train.py", line 200, in train
curr_test_metrics = test_epoch(model, device, val_loader,
File "/home/swufe1/project/Waveformer/src/training/eval.py", line 75, in test_epoch
tensorboard_add_metrics(
File "/home/swufe1/project/Waveformer/src/training/synthetic_dataset.py", line 162, in tensorboard_add_metrics
vals = np.asarray(metrics['scale_invariant_signal_noise_ratio'])
KeyError: 'scale_invariant_signal_noise_ratio'

Could you please provide any suggestions on how to resolve this issue? Thank you very much!

[000WSW]

Hello, I encountered the same issue while running the code. May I ask if you have solved this problem? Thank you very much.

[YuanxinGuo]

Sorry, I haven't resolved this issue yet. I modified line 162 of synthetic_dataset.py to vals = np.asarray(metrics['_scale_invariant_signal_noise_ratio']). The program successfully trains for one epoch, but during the second epoch, it still reports the error:
Test: 100%|█████████████████████████████████████████████████████████| 79/79 [08:12<00:00, 6.23s/it]
Test: _signal_noise_ratio=8.0362 _scale_invariant_signal_noise_ratio=3.4833 loss=-1.2454 runtime=82.2936
Traceback (most recent call last):
File "/home/swufe1/project/Waveformer/src/training/train.py", line 229, in train
if max(val_metrics[base_metric]) == val_metrics[base_metric][-1]:
KeyError: 'scale_invariant_signal_noise_ratio'

[000WSW]

Sorry, I haven't resolved this issue yet. I modified line 162 of synthetic_dataset.py to vals = np.asarray(metrics['_scale_invariant_signal_noise_ratio']). The program successfully trains for one epoch, but during the second epoch, it still reports the error: Test: 100%|█████████████████████████████████████████████████████████| 79/79 [08:12<00:00, 6.23s/it] Test: _signal_noise_ratio=8.0362 _scale_invariant_signal_noise_ratio=3.4833 loss=-1.2454 runtime=82.2936 Traceback (most recent call last): File "/home/swufe1/project/Waveformer/src/training/train.py", line 229, in train if max(val_metrics[base_metric]) == val_metrics[base_metric][-1]: KeyError: 'scale_invariant_signal_noise_ratio'

try set "base_metric": "_scale_invariant_signal_noise_ratio",in the config.json in the file of experiments

[YuanxinGuo]

I am sorry that I still encounter the same error after modifying the code according to your answer. Thanks again for your answer!