DRAM2 launches way to many SLURM jobs per input fasta

[madeline-scyphers]

Description of the bug

DRAM2 launches 1 job per input file for many steps in the pipeline (1 job per prodigal, 1 job per kegg search, 1 job per camper search, etc.). This does not scale at all with large numbers of mags or assemblies. We need to combine inputs, either by concatenating them together (after rename to make unique headers) or run things in for loops inside jobs (potentationly certain processes might be better one way or another). We obviously want to still partition things out to take advantage of the queue. And if concatenating, we probably want to try and make sure the concatenated file stays inside the provisioned memory, or we risk major slow downs.

System information

All versions of DRAM2 currently

Command used and terminal output

Relevant files

No response

[Gsmith535]

Hi Madeline!

Past DRAM contributor, currently trying to convince local microbiome folk of the value of DRAM and implement DRAM2 via bioinf core services. Just got DRAM2 to work on our cluster, mostly, and so I am looking to iron out kinks.

Can you clarify why this is a bug? To be honest, I would consider this a feature and would prefer this as, from my perspective using DRAM across several HPC systems. It is not efficient for most HPC users (I think) that each step is executed all in one job for a given genome due to vastly different resource requirements across the steps (e.g., prodigal vs KEGG search). Perhaps if steps with similar resource requirements were bundled, this could be useful for avoiding job limits.

Wouldn't this be managed using the --queue_size option to self-regulate? I apologize if this has already been solved.

[madeline-scyphers]

Hey Garret. Totally fair question. It isn't about reducing the ability to take advantage of HPCs high parallezation, it is more that with a large number of inputs, it can lead to thousands of jobs. 1 job per input for prodigal, 1 job per input per database search in annotate. 1 job per input for mmseqs index, trna, and rrna. Some databases have additional post processing steps as well. So if you are all the databases fully implemented right now (12 as of the update releasing today), that is 21* # of inputs + a few extra jobs for summarize and otehr things. So for 200 input files we are look at over 4000 slurm jobs. When DRAM was first ported to nextflow, there were a lot more processes that would launch jobs that would take 2 seconds - 30 seconds and then be done. Since I came on, I have collected a lot of those jobs, streamlining it as I go before I have time to really address this.

The problem is mostly that when you get to that many slurm jobs, busy slurm queues will never launch them fast enough, so it will take way longer because of it. If a slurm queue is busy from other users, then each job launch could have a delay in running, drastically slowing down DRAMs overall run time because of how long jobs spend pending.

So the approach is mostly moving certain processes that are tiny (take less than a 30 seconds or less or so) into other things as pre or post processing to it launches a lot less jobs, and then a larger update that will have some way of controlling batching jobs together. The two main ways we have talked about that is A) batching input files together so that instead of having say 200 input files and each one launch a job, you batch X together into bigger jobs, where X is a controllable parameter so people with wide open slurm queues can still launch as many as they can. B) They other possibility is putting things like the mmseqs and hmm searches together into the same job, so for a given input, one job does all mmseqs searchs or maybe all database searches together. Problem with that is that some databases require way different resource requests, so your job wouldn't be utilizing all its resources the whole time. Option C) is a combination of A and B.

Ideally whatever we do eventually, users can still run it in a way that really utilizes the HPCs queue and parralelism if they have an open queue, but gives users who have a busy queue and option to not get stuck in scheduling hell.

I hope that all makes sense.

[Gsmith535]

Madeline - Thanks for the detailed response. I completely understand and am interested in the possible solutions as i expect to be running DRAM2 frequently.
Curious about compatibility within a snakemake or broader nextflow framework.