Process multiple samples at once

[justsharan]

I have a bunch of reads in a folder like this:

SRR1610898_1.fastq
SRR1610898_2.fastq
SRR1610899_1.fastq
SRR1610899_2.fastq
SRR1610920_1.fastq 
SRR1610920_2.fastq
SRR1610921_1.fastq
SRR1610921_2.fastq
genome.fa
genes.gtf

and I have a pipeline:

params.outdir = "$projectDir/out"
params.reads  = "$projectDir/data/*_{1,2}.fastq"
params.genome = "$projectDir/data/genome.fa"

reads      = Channel.fromFilePairs(params.reads, checkIfExists: true)
reference  = Channel.fromPath(params.genome, checkIfExists: true)

workflow {
  BWA_INDEX(reference)
  BWA_ALIGNMENT(BWA_INDEX.out.bwa_index.first(), reads)
}

process BWA_INDEX {
    publishDir "${params.outdir}/bwa_index", mode:"copy"

    input:
    file(genome)

    output:
    tuple path(genome), path("*"), emit: bwa_index

    script:
    """
    bwa index $genome
    """
}

process BWA_ALIGNMENT {
    tag "$sample_id to reference"

    input:
    tuple path(genome), path("*")
    tuple val(sample_id), path(reads)

    output:
    "*.bam"

    script:
    """
    bwa mem -t $task.cpus $genome $reads \
        | samtools view -b \
        > ${sample_id}.bam
    """
}

As it stands, BWA_ALIGNMENT runs each sample after another, but I'd like it to run all 4 (or any number of samples) in parallel... how can I do this?

When I googled it, I found out about task batching, but I'm not sure how it would work if I need to pass in that other argument (the bwa_index one) as well.

I was also informed of maxForks when reading through past discussions, but when I tried adding maxForks 5 to the BWA_ALIGNMENT process, it changed nothing... it was still running the samples through the process one after another.

[mribeirodantas]

Nextflow will by default try to run them in parallel. If it's not doing that, it may be due to your OS, available resources and so on. Trying in a powerful machine with a workload manager may make it easier for you to realize in real scenario Nextflow making use of parallelization.

Task batching is kind of the opposite. Instead of having 20 tasks in parallel, you'd have 5, for example, as samples would be concentrated in fewer tasks.

[justsharan]

Got it! I realize I just misunderstood the output that Nextflow was showing me; the process was indeed running in parallel. Thanks for your help!