ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

[cutleraging]

Description of the bug

ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Seems to be when creating multiQC file? Perhaps this is due to wanting to create heatmap with all samples in deeptools?

Command used and terminal output

nextflow run nf-core/cutandrun \
-r 3.2.2 \
-profile singularity \
--input samplesheet.csv \
--outdir ./results \
--bowtie2 /gs/gsfs0/users/sidoli_Lab/References/Human-GRCh38.p13/Indexes/bowtie2_GRCh38 \
--gtf /gs/gsfs0/users/sidoli_Lab/References/Human-GRCh38.p13/Annotation/Gencode/gencode.v38.primary_assembly.annotation.gtf \
--blacklist /gs/gsfs0/users/sidoli_Lab/References/Human-GRCh38.p13/Annotation/hg38-blacklist.v2.bed \
--spikein_bowtie2 /gs/gsfs0/users/sidoli_Lab/References/Escherichia_coli_K-12_MG1655 \
--spikein_fasta /gs/gsfs0/users/sidoli_Lab/References/Escherichia_coli_K-12_MG1655/GCF_000005845.2_ASM584v2_genomic.fna \
--fasta /gs/gsfs0/users/sidoli_Lab/References/Human-GRCh38.p13/Genomes/GRCh38.primary_assembly.genome.fa \
--peakcaller seacr \
--replicate_threshold 2 \
--multiqc_title 2024-sarah-cutandtag

Relevant files

Archive 2.zip

System information

Nextflow version - version 24.04.4 build 5917
Hardware - HPC
Executor - slurm
Container engine - Singularity
OS - Linux
Version of nf-core/cutandrun - v3.2.2-g6e1125d

[KunFang93]

Hi,

I would like to report that I encountered same error with the error below:

ERROR ~ Error executing process > 'NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
  Process `NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS (1)` terminated with an error exit status (1)


Command executed:

  #!/usr/bin/env python

  import yaml
  import platform
  from textwrap import dedent

  def _make_versions_html(versions):
      html = [
          dedent(
              '''\
              <style>
              #nf-core-versions tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions">
                  <thead>
                      <tr>
                          <th> Process Name </th>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              '''
          )
      ]
      for process, tmp_versions in sorted(versions.items()):
          html.append("<tbody>")
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              html.append(
                  dedent(
                      f'''\
                      <tr>
                          <td><samp>{process if (i == 0) else ''}</samp></td>
                          <td><samp>{tool}</samp></td>
                          <td><samp>{version}</samp></td>
                      </tr>
                      '''
                  )
              )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)

  def _make_versions_unique_html(versions):
      unique_versions = []

      for process, tmp_versions in sorted(versions.items()):
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              tool_version = tool + "=" + version
              if tool_version not in unique_versions:
                  unique_versions.append(tool_version)

      unique_versions.sort()

      html = [
          dedent(
              '''\
              <style>
              #nf-core-versions-unique tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions-unique">
                  <thead>
                      <tr>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              '''
          )
      ]

      for tool_version in unique_versions:
          tool_version_split = tool_version.split('=')
          html.append("<tbody>")
          html.append(
              dedent(
                  f'''\
                  <tr>
                      <td><samp>{tool_version_split[0]}</samp></td>
                      <td><samp>{tool_version_split[1]}</samp></td>
                  </tr>
                  '''
              )
          )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)

  module_versions = {}
  module_versions["NFCORE_CUTANDRUN:CUTANDRUN:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
      'python': platform.python_version(),
      'yaml': yaml.__version__
  }

  with open("versions.yml") as f:
      workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions

  workflow_versions["Workflow"] = {
      "Nextflow": "24.04.4",
      "nf-core/cutandrun": "3.2.2"
  }

  versions_mqc = {
      'parent_id': 'software_versions',
      'parent_name': 'Software Versions',
      'parent_description': 'Details software versions used in the pipeline run',
      'id': 'software-versions-by-process',
      'section_name': 'nf-core/cutandrun software versions by process',
      'section_href': 'https://github.com/nf-core/cutandrun',
      'plot_type': 'html',
      'description': 'are collected at run time from the software output.',
      'data': _make_versions_html(workflow_versions)
  }

  versions_mqc_unique = {
      'parent_id': 'software_versions',
      'parent_name': 'Software Versions',
      'parent_description': 'Details software versions used in the pipeline run',
      'id': 'software-versions-unique',
      'section_name': 'nf-core/cutandrun Software Versions',
      'section_href': 'https://github.com/nf-core/cutandrun',
      'plot_type': 'html',
      'description': 'are collected at run time from the software output.',
      'data': _make_versions_unique_html(workflow_versions)
  }

  with open("software_versions.yml", 'w') as f:
      yaml.dump(workflow_versions, f, default_flow_style=False)

  with open("software_versions_mqc.yml", 'w') as f:
      yaml.dump(versions_mqc, f, default_flow_style=False)

  with open("software_versions_unique_mqc.yml", 'w') as f:
      yaml.dump(versions_mqc_unique, f, default_flow_style=False)

  with open('local_versions.yml', 'w') as f:
      yaml.dump(module_versions, f, default_flow_style=False)

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  Traceback (most recent call last):
    File ".command.sh", line 99, in <module>
      workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/__init__.py", line 114, in load
      return loader.get_single_data()
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
             ^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
      document = self.compose_document()
                 ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
      node = self.compose_node(None, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
      item_value = self.compose_node(node, item_key)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
      while not self.check_event(MappingEndEvent):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
      self.current_event = self.state()
                           ^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
      if self.check_token(KeyToken):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 116, in check_token
      self.fetch_more_tokens()
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
      return self.fetch_value()
             ^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 577, in fetch_value
      raise ScannerError(None, None,
  yaml.scanner.ScannerError: mapping values are not allowed here
    in "versions.yml", line 463, column 18
  INFO:    Cleaning up image...

Work dir:
  /data1/kfang/HuiZ/PancreaticCancer/rawdata/work/3f/82be7eb85508142fad6afac673f30a

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

And I cannot rollback to 3.6.3 as showed

(nextflow) [kfang@zjlabs rawdata]$ mamba install -c conda-forge "singularity=3.6.3"

Looking for: ['singularity=3.6.3']

conda-forge/noarch                                  16.0MB @  24.7MB/s  0.7s
conda-forge/linux-64                                37.0MB @  40.5MB/s  1.0s

Pinned packages:
  - python 3.12.*


warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
Could not solve for environment specs
The following packages are incompatible
├─ python is installable with the potential options
│  ├─ python 3.12.0 would require
│  │  └─ openssl >=3.1.3,<4.0a0 , which can be installed;
│  ├─ python [3.12.4|3.12.5] would require
│  │  └─ openssl >=3.3.1,<4.0a0 , which can be installed;
│  ├─ python [1.0.1|1.2|...|3.9.9], which can be installed;
│  ├─ python 3.12.0rc3 would require
│  │  └─ _python_rc, which does not exist (perhaps a missing channel);
│  ├─ python 3.12.1 would require
│  │  └─ openssl >=3.2.0,<4.0a0 , which can be installed;
│  └─ python [3.12.2|3.12.3] would require
│     └─ openssl >=3.2.1,<4.0a0 , which can be installed;
└─ singularity 3.6.3**  is not installable because it requires
   └─ openssl >=1.1.1g,<1.1.2a , which conflicts with any installable versions previously reported.

I wondered if you could shed some light on this? Thanks a lot for helping!

Best,
Kun

[KunFang93]

I worked around it with downgrading singularity to 3.6.3 and used -r 3.2.1

[chris-cheshire]

Hi sorry for the delay, this is weird! I use singularity 3.6.4, and I dont get this error.

[KunFang93]

Yes..It's weird, but my singularity version is 3.8.6 before downgrade to 3.6.3. Conda cannot solve 3.6.4 for our server

I checked versions.yml and looks like the problem occurs on

"NFCORE_CUTANDRUN:CUTANDRUN:ALIGN_BOWTIE2:BOWTIE2_TARGET_ALIGN":
    bowtie2: perl:
    samtools: samtools: error while loading shared libraries: libhts.so.3: cannot open shared object file: No such file or directory
    pigz: pigz: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

Hope this helps...

Kun