using mpirun in a HPC cluster's node: SLURM process starter for OpenMPI was unable to locate a usable "srun" command in its path, hostname vs /etc/hosts

[SomePersonSomeWhereInTheWorld]

$ singularity --version 
singularity version 3.7.1

Describe the bug

./runSnappyHexMesh.sh
Run snappyHexMesh in parallel on 8 processors
--------------------------------------------------------------------------
The SLURM process starter for OpenMPI was unable to locate a
usable "srun" command in its path. Please check your path
and try again.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
An internal error has occurred in ORTE:

[[40700,0],0] FORCE-TERMINATE AT (null):1 - error plm_slurm_module.c(471)

This is something that should be reported to the developers.
--------------------------------------------------------------------------

To Reproduce
Steps to reproduce the behavior:
Using an OpenFOAM container when running a script, e.g., runSnappyHexMesh.sh which contains:

    echo -e "Run snappyHexMesh in parallel on $NP processors"
    mpirun -np $NP snappyHexMesh -overwrite -parallel  > snappyHexMesh.out

This fails with the above error. Based on this solution over at the OpenMPI Github, it happens when the value of hostname is not in /etc/hosts. This is the case in our HPC cluster on compute/execute nodes.

Expected behavior
Is there a way to work around this or set some environment variable?

OS / Linux Distribution
Which Linux distribution are you using?

$ cat /etc/os-release
RHEL 8.2

Installation Method
Via source

[dtrudg]

Hi - I've transferred this to a discussion, as it's not really a bug with Singularity, but a usage issue on a certain setup.

First - the Singularity version in use (3.7.1) is now very old. I'd suggest updating to 3.11.3 if at all possible.

This fails with the above error. Based on open-mpi/ompi#8554 (comment), it happens when the value of hostname is not in /etc/hosts. This is the case in our HPC cluster on compute/execute nodes.

AFAIK the issue you link is not specifically /etc/hosts related. If you hostnames are in DNS that is configured on them then it should work without an entry in /etc/hosts. Can you ping your host by its hostname? If not, that is going to cause general issues for multi-node jobs.

If your hostname is resolvable it may, in fact, be a non srun on PATH issue here. I'd suggest providing a bit more information about your SLURM configuration, MPI stack, and exactly what container you are using. It's important to know exactly how singularity is invoked, which isn't visible in the detail you provided. The more information you can provide, the more likely someone might spot the issue.

[SomePersonSomeWhereInTheWorld]

Can you ping your host by its hostname?

Yes.

If your hostname is resolvable it may, in fact, be a non srun on PATH issue.

It's in /cm/shared/apps/slurm/current/bin/srun. Or what do you mean by "non srun"?

FWIW, when I did add the hostname to /etc/hosts, the command works fine.

I'd suggest providing a bit more information about your SLURM configuration, MPI stack, and exactly what container you are using. It's important to know exactly how singularity is invoked, which isn't visible in the detail you provided.

Slurm 20.02.6 running via Bright Computing (now owned by Nvidia) Cluster on RHEL 8, OpenMPI compiled from source. I take the Docker file from GeoChemFoam via singularity pull and open a shell with singularity -v shell --bind /path/to/my/runs:/home/gcfoam/works/GeoChemFoam-5/runs:rw geocheme5.0.sif. Using the Test Case 01 Species transport, which does this command: mpirun -np $NP snappyHexMesh -overwrite -parallel > snappyHexMesh.out

[dtrudg]

Can you ping your host by its hostname?

Yes.
...
FWIW, when I did add the hostname to /etc/hosts, the command works fine.

If you can ping the hostname from the host shell, then either it's resolvable with DNS, or via the host's /etc/hosts file, so it should be resolvable in the container...

By default, Singularity has a bind path = /etc/hosts entry in its singularity.conf configuration file. It also has config resolv_conf = yes set. Together, these mean that the host's /etc/hosts file will be available in the container, and that the /etc/resolv.conf in the container will be set to use the DNS server that's configured on the host. This should replicate the resolution of the hostname that succeeds when you ping it outside of the container.

Have either of these options been disabled in singularity.conf?

If not...

• Try updating to a newer version of SingularityCE. 3.7.1 is very old and unsupported. It's possible a bug that has been fixed that might be impacting the behaviour here.
• Note any changes that have been made to singularity.conf from the defaults.
• On a current version of Singularity (i.e. 3.11), provide debug output of container setup (singularity debug shell ....), and the contents of /etc/hosts and /etc/resolv.conf both inside and outside of the container.
• Provide the output of hostname -A both on the host, and inside the unmodified container.

We also really need a more complete description of how the container is run / what it contains, as it doesn't appear as though execution is using the common hybrid or bind workflows (https://docs.sylabs.io/guides/latest/user-guide/mpi.html). In both of these models, mpirun is called on the host. I.E. mpirun singularity ... and not inside the container.

• It looks like you are using mpirun in the container? What variant / version is it?
• How is your Slurm job started, in which you run singularity shell?
• What PMIx / PMI / Slurm support is enabled in the container's MPI implementation?

[SomePersonSomeWhereInTheWorld]

By default, Singularity has a bind path = /etc/hosts entry in its singularity.conf configuration file. It also has config resolv_conf = yes set. Together, these mean that the host's /etc/hosts file will be available in the container, and that the /etc/resolv.conf in the container will be set to use the DNS server that's configured on the host. This should replicate the resolution of the hostname that succeeds when you ping it outside of the container.

Have either of these options been disabled in singularity.conf?

No both are there

config resolv_conf = yes
bind path = /etc/localtime
bind path = /etc/hosts

• Note any changes that have been made to singularity.conf from the defaults.
  Default.

• Provide the output of hostname -A both on the host, and inside the unmodified container.

Same in both:

Singularity> hostname -A
node029.cm.cluster node029.ib.cluster 
Singularity> exit
exit
[me@node029 runs]$ hostname -A
node029.cm.cluster nide029.ib.cluster 

We also really need a more complete description of how the container is run / what it contains, as it doesn't appear as though execution is using the common hybrid or bind workflows (https://docs.sylabs.io/guides/latest/user-guide/mpi.html). In both of these models, mpirun is called on the host. I.E. mpirun singularity ... and not inside the container.

Yeah we run mpirun inside the container and the scripts are described in this tutorial.
singularity -v shell --bind /path/to/my/runs:/home/gcfoam/works/GeoChemFoam-5/runs:rw ../geocheme5.0.sif

• It looks like you are using mpirun in the container? What variant / version is it?

mpirun (Open MPI) 4.1.5a1

• How is your Slurm job started, in which you run singularity shell?

srun --pty -t 0-08:00 --ntasks 8 --mem=10gb -X -A <my-account> /bin/bash

• What PMIx / PMI / Slurm support is enabled in the container's MPI implementation?

Is this enough?
gcc -I/usr/mpi/gcc/openmpi-4.1.5a1/include -I/usr/mpi/gcc/openmpi-4.1.5a1/include/openmpi -I/usr/mpi/gcc/openmpi-4.1.5a1/include/openmpi/opal/mca/hwloc/hwloc201/hwloc/include -pthread -L/usr/mpi/gcc/openmpi-4.1.5a1/lib64 -Wl,-rpath -Wl,/usr/mpi/gcc/openmpi-4.1.5a1/lib64 -Wl,--enable-new-dtags -lmpi

And actually going back to your original comment:

If your hostname is resolvable it may, in fact, be a non srun on PATH issue here.

Lo and behold I think I have the solution but perhaps you may have another. After loading the modules, singularity, openmpi and Python I export SINGULARITYENV_PREPEND_PATH=$PATH

Now that allows for srun to be seen in the container and mpirun runs as expected.

[dtrudg]

We've definitely ruled out it being a hostname / DNS resolution issue, then.

Lo and behold I think I have the solution but perhaps you may have another. After loading the modules, singularity, openmpi and Python I export SINGULARITYENV_PREPEND_PATH=$PATH

Hmmm, if injecting host PATH into the container makes MPI things work, then it usually means your Singularity configuration, or env vars in modules, are set to bind openmpi / slurm related binaries and libraries into the container by default.

Are you sure there are no additional bind path entries in singualrity.conf, nor SINGULARITY_BIND / SINGULARITY_BINDPATH env vars set via environment modules?

Anyway, glad you find a solution. As a general rule for MPI related problems, the simpler the example you can provide, the more chance there is someone can actually reproduce themselves. With the type of setup here... a large complex container with wrapper scripts etc. it's not really possible to spend time diving in to reproduce.

I'd also advise moving the mpirun invocation outside of the container, wherever possible,.... so any initial Scheduler<->MPI wire-up isn't crossing container / host boundaries. This is often a source of issues.

[SomePersonSomeWhereInTheWorld]

We've definitely ruled out it being a hostname / DNS resolution issue, then.

I'm still wondering why adding the hostname into /etc/hosts also made mpirun work.

Lo and behold I think I have the solution but perhaps you may have another. After loading the modules, singularity, openmpi and Python I export SINGULARITYENV_PREPEND_PATH=$PATH

Hmmm, if injecting host PATH into the container makes MPI things work, then it usually means your Singularity configuration, or env vars in modules, are set to bind openmpi / slurm related binaries and libraries into the container by default.

Are you sure there are no additional bind path entries in singualrity.conf, nor SINGULARITY_BIND / SINGULARITY_BINDPATH env vars set via environment modules?

In the modulefile there is:
prepend-path SINGULARITY_BINDPATH /storage,/cm,/local

And these in the conf file:

bind path = /etc/localtime
bind path = /etc/hosts
user bind control = yes

I'd also advise moving the mpirun invocation outside of the container, wherever possible,.... so any initial Scheduler<->MPI wire-up isn't crossing container / host boundaries. This is often a source of issues.

The issue there is that the mpirun commands are within scripts that are in the container as noted in this tutorial. We'd have to pull the commands out of the scripts I suppose.

[dtrudg]

Ahh, mounting all of the /cm tree, so that modules come into a container is often going to cause problems. Generally you want to avoid mixing host binaries and libraries into the container. There's no guarantee that software built for the host, under /cm, is compatible with a given container.

Apart from specific bindings of MPI libraries in the binding scenario that's in the user-guide, it's best to use only libraries and binaries provided by the container inside the container. The container is a separate environment from the host, that should provide everything needed to run the contained software.

At the very least, mixing host -> container will cause trouble where e.g. the container is built from a distro using a glibc version incompatible with the host, and the apps bound in from it.

I'm afraid that with /cm mounted, modules in use, and different mixes of host and container MPI possibly in play (depending on PATH, LD_LIBRARY_PATH), plus the complication of scripts driving MPI startup in the container... it's going to be difficult to reason what exactly is happening here without actually being on your system. There is a huge dependency on the state of the host, what modules are loaded, whether PATH propogates into the container etc.

Given the way this software is packaged in a container, it's likely that any other questions / issues would be better addressed in the user community for the software.