CUDA error, related to multiprocessing

[cbaakman]

I'm tried to run LightMHC with Cuda 11.8.0

Command:

python LightMHC/lightmhc/inference.py data.input_csv_path=xray.lightmhc.csv data.output_dir=lighmhc-xray model.n_cpus=32 model.use_gpu=true model.batch_size=64

I got an error message:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/baakmanc/LightMHC/lightmhc/inference.py", line 77, in workflow
    model = model.to(device)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 673, in to
    return self._apply(convert)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 409, in _apply
    param_applied = fn(param)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 671, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/home/baakmanc/miniconda3/envs/lightmhc/lib/python3.8/site-packages/torch/cuda/__init__.py", line 160, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
"""

I found that the start method can be set to 'spawn' by following the instructions here:
https://pytorch.org/docs/stable/notes/multiprocessing.html

That fixed the issue for me. Just letting you know.

[antoine-delaunay]

@cbaakman Thanks for spotting this, we will have a look into this and update our code accordingly