Maximizing Performance on HPC Falcon

[crswong888]

Hello, I am trying to understand how mpiexec -n <num_procs> and --n-threads=<num_threads> are translated when submitting jobs on INL HPC through the PBS system.

One thing is clear to me: num_procs is set by -l select=<num_nodes>:ncpus=<num_cores>:mpiprocs=<num_procs>.

However, I'm not sure how shared memory works on Falcon. Particularly, what the OMP_NUM_THREADS environment variable does. If I don't set it, than the default is num_cores, which I think implies 1 thread/core, which is generally what I want. However, the MOOSE job script example on the HPC website (https://hpcweb.hpc.inl.gov/software/open_speedshop/) sets OMP_NUM_THREADS=1 even though ncpus=24:mpiprocs=8, so this confuses me.

One other thing I'm concerned about is the MV2_ENABLE_AFFINITY environment variable. If this variable is enabled (default), I get a warning about slow performance when OMP_NUM_THREADS is non-zero (meanwhile zero isn't even a valid input). I read online that you generally want to disable affinity when hyper-threading so that you don't accidentally send too many threads to any one proc. But I also read that enabling affinity is great for single-threaded processes, which again, is generally what I want.

I guess my question is the following: How should I set the OMP_NUM_THREADS and MV2_ENABLE_AFFINITY environment variables if I don't want to use shared memory, i.e., if I want --n-threads=1, when running a MOOSE app on Falcon cluster? My guess is OMP_NUM_THREADS=<num_procs> and MV2_ENABLE_AFFINITY=0

[GiudGiud]

For full MPI and no shared memory parallelism (1 thread), I would do OMP_NUM_THREADS=1 (does not matter though, --n-threads should have the final say on the number of threads, and 1 is the default anyway) and MV2_ENABLE_AFFINITY=1, which is recommended for single threaded applications.

Guillaume

[crswong888]

Okay, I will do this then. Thank you. So if MV2_ENABLE_AFFINITY=1, then I always get a warning. But you're saying it's fine, so it should be safe to turn the warning off, yes?

Also, OMP_NUM_THREADS=1 is not the default. The default is the number of cores that are available, which is however many you request.

[GiudGiud]

But MOOSE does not use OMP_NUM_THREADS to determine its number of threads right?

The warning probably comes from Falcon, which detects the two environment variables that are somewhat incompatible.
Or does it come from MOOSE?

[crswong888]

No it definitely comes from Falcon, but I just want to be sure that everything I'm setting maximizes performance (hard to test since I have to wait in line every time I change something).

And yes, OMP_NUM_THREADS doesn't seem to set --n-threads, as indicated by the terminal printout at the beginning of the solve. But I'm just concerned that it still has an effect on performance. Anyways, I'll just go with your original suggestion. It seems like the right way to go.