Brief notes from building and running things on an HPC cluster

[rcarson3]

So, I took a shot at running/building AdditiveFoam on one of the HPC systems at LLNL that's a TOSS4 (RHEL8-based) system.

Thanks to @colemanjs help I was able to get everything built and able to get the tests up and running.

I just wanted to note a few quick things that could be helpful for other users.

While OpenFoam claims you need quite a few things such as paraview, I did not find that necessary to get things running. You only need their TPLs located at this directory: https://github.com/OpenFOAM/ThirdParty-10

For OpenFoam, you'll want to modify their OpenFoam/etc/bashrc file. @colemanjs provided me the following info that was helpful to get up and running with MPICH/MVAPICH and GCC code.

Around the WM_MPLIB line you can modify things to:

export MPI_ARCH_FLAGS="-DMPICH_SKIP_MPICXX"
export MPI_ARCH_INC="-I$MPI_ROOT/include"
export MPI_ARCH_LIBS="-L$MPI_ROOT/lib -lmpich -lrt"
export WM_MPLIB=SYSTEMMPI

and then set the following line in that same file:

export FOAM_SIGFPE=unset

Next, running OpenFOAM ./Allwmake just build things in serial. You'll definitely want to remember to add that -j flag in there for parallel builds.

For AdditiveFoam, if you're running the examples/tutorials depending on how your HPC system is set-up you might have to modify the OpenFoam/bin/tools/RunFunctions file where it uses mpirun and sub it out for your systems appropriate job queue run commands such as srun on a SLURM job queue system or jsrun on a LSF job queue system.