Disable task parallelism

[raygozag]

Hi, I'm running a predefined pipeline and one of the steps is by far the most time consuming in terms of disk access, especially in a cluster with network file systems.

However, when running a large dataset the pipeline chokes trying to run other tasks for some of the finished elements in this intermediate task, which would actually run faster if one let this one task finish for all the samples, and then run the rest however it may be.

is there a way as a user or to modify the pipeline to specify that a no tasks should be started ahead until all the samples have been finished processing in this one first?

[Subject-13]

Hey, there are probably multiple solutions for that. You can either limit the resources of the most time consuming process by setting a process cpus directive like this, so the other processes won't choke as you describe (if they also rely on multiple cores). So each process of timeEater won't use more than 4 cores, if available. Additionally you can set a maxForks directive to limit the maximum parallel running processes of timeEater. In this example, no more than two processes of timeEater would run simultaneously and each process would not use more than 4 cores, if these are available at all:

process timeEater {
   cpus 4
   maxForks 2

   script:
   """
   something
   """
}

If you want that your process shall finish all its tasks first before the following processes start, you can use the collect() operator for the following process. In this way process B waits for all outs of process A, no matter how many tasks process A started. But be aware the collect() operator collects all items from a source channel into a list and emits it as a single item. Therefore the flatten() operator should flattens each item from a list or other collection, such that each element in each collection is emitted separately to your next process. It could look like this:

process A {
   ...
}

process B {
   ...
}

workflow {
   A(inputs)

   B(A.out.collect().flatten())
}

I didn't try that out, but my guess is it should behave like you described.

EDIT: Typos

[colindaven]

Another useful nextflow.config option to limit the number of tasks submitted is

https://www.nextflow.io/docs/latest/config.html

eg
executor.queueSize = 2