Multi-GPUs not working

[hosseinbv]

Hi,
The multi-gpu setting does not work.

return forward_call(*input, **kwargs)

File "/home/hossein/projects/Ladder-Side-Tuning-main/seq2seq/third_party/models/t5/modeling_side_t5.py", line 1219, in forward
side_gate_param = self.side_gate_params[i]
File "/media/data/hossein/envs/ladder/lib/python3.7/site-packages/torch/nn/modules/container.py", line 462, in getitem
idx = self._get_abs_string_index(idx)
File "/media/data/hossein/envs/ladder/lib/python3.7/site-packages/torch/nn/modules/container.py", line 445, in _get_abs_string_index
raise IndexError('index {} is out of range'.format(idx))
IndexError: index 1 is out of range

[ylsung]

Hi @hosseinbv, thank you for pointing out the issue.

I didn't try the multi-GPU training in my experiments, so there might be some problems. Feel free to send PR if you fix the problem. If you want to run the program, maybe try a smaller batch size and adjust the learning rate accordingly. Another option is accumulating gradients over k batches to mimic a larger batch.