Warn: Failed to publish file

[zillurbmb51]

Hi, I have a process as below and I am getting Warn: Failed to publish file But I have write permission to the publish directory.

process merging {
publishDir params.out, mode:'copy', overwrite: true
input:
val flag from done_ch
tuple val(chrom),path(bgens) from imputed
output:
tuple val("${chrom}"),path("${chrom}.*") into merged
script:
bgen_files = bgens.collect().findAll { it.toString().endsWith('.bgen') }.sort()
gen_files = bgens.collect().findAll { it.toString().endsWith('.gen.gz') }.sort()
sample_files = bgens.collect().findAll { it.toString().endsWith('.sample') }.sort()
stat_files = bgens.collect().findAll { it.toString().endsWith('.stats') }.sort()
"""
echo "${bgen_files.join(' ')}" > "${chrom}.step10.imputed.merged.bgen.txt"
echo "${gen_files.join(' ')}" > "${chrom}.step10.imputed.merged.gen.gz.txt"
echo "${sample_files.join(' ')}" > "${chrom}.step10.imputed.merged.sample.txt"
echo "${stat_files.join(' ')}" > "${chrom}.step10.imputed.merged.snp.stats.txt"
"""
}

nextflow.log.txt

[abhi18av]

Hi @zillurbmb51 ,

Based on the logs, it seems that this is due to a non-existant file.

Jul-02 12:08:54.425 [FileTransfer-thread-3] WARN  nextflow.processor.PublishDir - Failed to publish file: /gpfs/projects02/p697/cluster/projects/moba_qc_imputation/ZR/testnextflow/gsa/work/51/ce3fb422f34d80b73b05c530ec9a3b/chr1.step10.imputed.merged.gen.gz.txt; to: /gpfs/projects02/p697/cluster/projects/moba_qc_imputation/ZR/testnextflow/gsa/rujescu_out_chunk/chr1.step10.imputed.merged.gen.gz.txt [copy] -- See log file for details
java.nio.file.NoSuchFileException: /gpfs/projects02/p697/cluster/projects/moba_qc_imputation/ZR/testnextflow/gsa/rujescu_out_chunk/chr1.step10.imputed.merged.gen.gz.txt

[zillurbmb51]

Thanks. But I do not understand what does it mean.
Also in the "${chrom}.step10.imputed.merged*txt" There is only the last entry from the echo command.
How could I get all the file names from echo in the "${chrom}.step10.imputed.merged.snp.stats.txt"?

[abhi18av]

Basically, you need to confirm whether the file /gpfs/projects02/p697/cluster/projects/moba_qc_imputation/ZR/testnextflow/gsa/rujescu_out_chunk/chr1.step10.imputed.merged.gen.gz.txt exists in your filesystem or not ?

How could I get all the file names from echo in the "${chrom}.step10.imputed.merged.snp.stats.txt"?

Not sure what you mean here, could you please elaborate a bit?

[zillurbmb51]

Thanks a lot. The files exist in here: /cluster/projects/p697/projects/moba_qc_imputation/ZR/testnextflow/gsa/rujescu_out_chunk/chr1.step10.imputed.merged.gen.gz.txt, I am not sure how it gets /gpfs/projects02/

imputed is a output channel of process imputation where I have output of tuples (value, paths) here value=chr1,chr2,chr3......chr22 and path=path of multiple outputs with different extension (bgen, gz,stats,sample). such as; [chr1,[[chr1.chunk1.bgen],[chr1.chunk1.stats]], [chr1,[[chr1.chunk2.bgen],[chr1.chunk2.stats]],[chr1,[[chr1.chunk3.bgen],[chr1.chunk3.stats]]...

process imputation {
publishDir params.out, mode:'copy'
input:
tuple val(chrom),val(chunk_array),val(chunk_start),val(chunk_end),path(in_haps),path(refs),path(maps) from imp_ch
output:
tuple val("${chrom}"),path("${chrom}.*") into imputed
script:
def (haps,sample)=in_haps
def (haplotype, legend, samples)=refs
"""
impute4.1.2_r300.3 -g "${haps}" -h "${haplotype}" -l "${legend}" -m "${maps}" -o "${chrom}.step10.imputed.chunk${chunk_array}" -no_maf_align -o_gz -int "${chunk_start}" "${chunk_end}" -Ne 20000 -buffer 1000 -seed 54321

if [[ \$(gunzip -c "${chrom}.step10.imputed.chunk${chunk_array}.gen.gz" | head -c1 | wc -c) == "0" ]]
then 
 echo  "${chrom}.step10.imputed.chunk${chunk_array}.gen.gz" is empty
else
 qctool_v2.0.8_rhel -g "${chrom}.step10.imputed.chunk${chunk_array}.gen.gz" -snp-stats -osnp "${chrom}.step10.imputed.chunk${chunk_array}.snp.stats"
 qctool_v2.0.8_rhel -g "${chrom}.step10.imputed.chunk${chunk_array}.gen.gz" -og "${chrom}.step10.imputed.chunk${chunk_array}.bgen" -os "${chrom}.step10.imputed.chunk${chunk_array}.sample"
fi
 """

I wanted to create a list of chunks for respective chr with same extension. The above code only gives me the last one in the "${chrom}.step10.imputed.merged*txt". For example, I have 77 chunks (files) for chr1. So, from the above echo command, I would expect a file with 77 lines, but I am getting only one line with the 77th file/chunk name. Not sure if my explanation clarify the issue.

[abhi18av]

I am not sure how it gets /gpfs/projects02/

I'm guessing that this might have to do with the resolution of baseDir or projectDir somewhere else in the code, if it's not a typo.

from the above echo command, I would expect a file with 77 lines, but I am getting only one line with the 77th file/chunk name.

What I understand is that you would like to have all files within this process, in that case you need to use collect operator on the input channel to receive and operate on all of these files.

[zillurbmb51]

Thank you. So, How could I get rid of those warnings (Failed to publish file)?

I think I used collect operator but no improvement in the output.

script:
bgen_files = bgens.collect().findAll { it.toString().endsWith('.bgen') }.sort()
gen_files = bgens.collect().findAll { it.toString().endsWith('.gen.gz') }.sort()
sample_files = bgens.collect().findAll { it.toString().endsWith('.sample') }.sort()
stat_files = bgens.collect().findAll { it.toString().endsWith('.stats') }.sort() 

I also tried script:
bgen_files = bgens.collect({it}).findAll { it.toString().endsWith('.bgen') }.sort()
But got same result. Not sure what I am missing here.

[zillurbmb51]

Thanks for helping. I tried from the beginning and come up with the following solution.

imputed.into{impute_bgen;impute_gen;impute_sample;impute_stat}
bgens=impute_bgen.groupTuple().transpose().map{chrom,bfiles -> tuple(chrom,bfiles[0])}.groupTuple()
gens=impute_gen.groupTuple().transpose().map{chrom,bfiles -> tuple(chrom,bfiles[1])}.groupTuple()
samples=impute_sample.groupTuple().transpose().map{chrom,bfiles -> tuple(chrom,bfiles[2])}.groupTuple()
stats=impute_stat.groupTuple().transpose().map{chrom,bfiles -> tuple(chrom,bfiles[3])}.groupTuple()