The distributed version is sending a copy of the whole dataset to each parallel worker:
|
callInternalCoGAPS <- function(data, allParams, uncertainty, subsetIndices, |
and the lapply loop:
|
callInternalCoGAPS(data, allParams, uncertainty, sets[[i]], i) |
On slurm servers this ends up with a out-of-memory error:
slurmstepd: error: Detected 3 oom_kill events in StepId=19656397.batch. Some of the step tasks have been OOM Killed.
Let's subset the data for each worker to reduce the memory footprint.
However, there is an important exception: If data is a file path (character string), only the filename is sent to each worker, and each worker reads from the same file. So for file-based inputs we may want to add row/column range parameters to cogaps_from_file_cpp so workers read only their subset directly from disk.
Uncertainty also needs to be subset if it's provided.
The distributed version is sending a copy of the whole dataset to each parallel worker:
CoGAPS/R/DistributedCogaps.R
Line 12 in 3b3e002
and the lapply loop:
CoGAPS/R/DistributedCogaps.R
Line 71 in 3b3e002
On slurm servers this ends up with a out-of-memory error:
Let's subset the data for each worker to reduce the memory footprint.
However, there is an important exception: If data is a file path (character string), only the filename is sent to each worker, and each worker reads from the same file. So for file-based inputs we may want to add row/column range parameters to cogaps_from_file_cpp so workers read only their subset directly from disk.
Uncertainty also needs to be subset if it's provided.