MOOSE on copenhagen university HPC

[AndreasHjortsoe]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

Issue description

I am trying to install MOOSE on a HPC cluster in copenhagen.
I initialize with:

• module load mpi/mpich-x86_64
• export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77
• export HDF5_DIR=/usr (i tried fixing my issues with this, in order to make sure HDF5 was the correct version, but to no avail)

When i run "./update_and_rebuild_petsc.sh":

• I have added " CC=$CC CXX=$CXX FC=$FC F77=$F77 F90=$F90 " to the configure line in configure_petc.sh
• There might be issues with HDF5, and this may be the cause. This is the reason i added export HDF5_DIR.

When i run "./update_and_rebuild_libmesh.sh":

i got errors, showing that there was no available MPI_Comm, indicating that there are issues with libmesh finding my MPI.

• I fixed this by adding "--with-mpi-include="/usr/include/mpich-x86_64" to the configure line in moose/scripts/configure_libmesh.sh

Now i can run "./update_and_rebuild_wasp.sh", and make without errors.

Running tests

After running make, i tried ./run_tests, which yeilded 51/51 failed tests. Can anyone help with troubleshooting please?

Output from running ./run_tests -i always_bad: i am unsure why it takes so long to run. Yesterday it ran much faster, but today it is slow. Yesterday i got CODE 1 error for both, instead of the TIMEOUT error.

[hjortso@hep03 test]$ ./run_tests -i always_bad
tests/test_harness.always_ok ......................................................................... RUNNING
tests/test_harness.always_ok: Working Directory: /lustre/hpc/hep/hjortso/temp_dir/moose/test/tests/test_harness
tests/test_harness.always_ok: Running command: /lustre/hpc/hep/hjortso/temp_dir/moose/test/moose_test-opt -i good.i --error --error-override --libtorch-device cpu
tests/test_harness.always_ok:
tests/test_harness.always_ok ..................................................... [FINISHED] FAILED (TIMEOUT)
tests/test_harness.always_bad: Working Directory: /lustre/hpc/hep/hjortso/temp_dir/moose/test/tests/test_harness
tests/test_harness.always_bad: Running command: false
tests/test_harness.always_bad:
tests/test_harness.always_bad: ################################################################################
tests/test_harness.always_bad: Tester failed, reason: CODE 1
tests/test_harness.always_bad:
tests/test_harness.always_bad ................................................................ FAILED (CODE 1)


Final Test Results:
--------------------------------------------------------------------------------------------------------------
tests/test_harness.always_ok ..................................................... [FINISHED] FAILED (TIMEOUT)
tests/test_harness.always_bad ................................................................ FAILED (CODE 1)
--------------------------------------------------------------------------------------------------------------
Ran 2 tests in 571.9 seconds. Average test time 150.0 seconds, maximum test time 300.0 seconds.
0 passed, 0 skipped, 0 pending, 2 FAILED

Diagnostics output - i see nothing out of the ordinary here, but there might be

####################################################################################################
Influential Environment Variables

CC=mpicc
CONDA_CHANNEL=https://conda.software.inl.gov/public
CURL_CA_BUNDLE=
CXX=mpicxx
F77=mpif77
F90=mpif90
FC=mpif90
FI_PROVIDER=tcp
HDF5_DIR=/usr
LD_LIBRARY_PATH=/usr/lib64/mpich/lib
MODULESHOME=/usr/share/lmod/lmod
MOOSE_JOBS=6
PATH=/usr/lib64/mpich/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/groups/hep/hjortso/bin
REQUESTS_CA_BUNDLE=
SSL_CERT_FILE=
####################################################################################################
Compiler(s) (CC CXX FC F77 F90):

which $CC; /usr/lib64/mpich/bin/mpicc
$CC --version; gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
$CC -show; gcc -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -I/usr/include/mpich-x86_64 -L/usr/lib64/mpich/lib -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

which $CXX; /usr/lib64/mpich/bin/mpicxx
$CXX --version; g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
$CXX -show; g++ -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -I/usr/include/mpich-x86_64 -L/usr/lib64/mpich/lib -lmpicxx -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

which $FC; /usr/lib64/mpich/bin/mpif90
$FC --version; GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
$FC -show; gfortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -I/usr/include/mpich-x86_64 -I/usr/lib64/gfortran/modules/mpich -L/usr/lib64/mpich/lib -lmpifort -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

which $F77; /usr/lib64/mpich/bin/mpif77
$F77 --version; GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
$F77 -show; gfortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -fallow-argument-mismatch -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -I/usr/include/mpich-x86_64 -I/usr/lib64/gfortran/modules/mpich -L/usr/lib64/mpich/lib -lmpifort -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

which $F90; /usr/lib64/mpich/bin/mpif90
$F90 --version; GNU Fortran (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
$F90 -show; gfortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -I/usr/include/mpich-x86_64 -I/usr/lib64/gfortran/modules/mpich -L/usr/lib64/mpich/lib -lmpifort -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags -lmpi

OK
####################################################################################################
Python Sanity Checks

/usr/bin/env python3 --version; (reporting as: Python 3.9.18) matches
which python3 python;

/usr/bin/python3 --version; == Python 3.9.18
/usr/bin/python --version; == Python 3.9.18

OK
####################################################################################################
Python Modules (TestHarness, run-ability)

OK

checks PASSED

No response

[GiudGiud]

Hello

Can you please attach the logs for building petsc (if you think there were issues), libmesh (there is a configure.log) that should help and moose?

[GiudGiud]

also what does ldd moose_test-opt return ?

[AndreasHjortsoe]

petcs

Petsc is building fine, and running the "make test" commands stated at the end of the "update_and_rebuild_petcs" file work without errors, so petcs probably works, but i have added the configure.log anyways as configure(petcs).
configure(petsc).log

libmesh

I found the output of configure from libmesh in here (libmesh_diagnostic.log):
libmesh_diagnostic.log

MOOSE

I have now run ./configure in the moose directory to generate the configure file for moose:
config(moose).log
I found config.status as well, maybe that is usefull for you:
config(moose)txt.txt

moose_test yeilds a segmentation error, and ldd_moose_test is below:
ldd_moose_test.out.txt

Thanks for taking a look, i hope this is sufficient! Otherwise please let me know what you need.

Regards Andreas

[GiudGiud]

Nothing obvious to me.
Can you run an input in a debugger and get us a backtrace for the error?
This is a recent version of moose right?
https://mooseframework.inl.gov/application_development/debugging.html

[AndreasHjortsoe]

Okay, here it is. It looks like the error is in libmesh, in netgen?

debug_info.txt

I believe the version is the most recent one yes. I have done the steps from "common installation issues (#27046) "

[GiudGiud]

Yes it does look like it's coming from netgen.

So for now, I recommend you disable netgen by passing --disable-netgen to update_and_rebuild_libmesh.sh

You ll have to check the log to check if it actually worked because I seem to remember the option was not listed in the help page though it should be working still

I ll take care of creating an issue so this can be looked at

[mangerij]

Hi @AndreasHjortsoe

I'm actually local up at DTU. Are you using Niflheim? If so, I can probably help out.

[AndreasHjortsoe]

No, unfortunately i am on a small local HPC here at KU