Petsc Has Not Been Detected

[dorton21]

For this instance, I used the offline instructions to setup and install moose. When I go to run the test Makefile the following error is given:

"#error PETSc has not been detected, please ensure your environment is set up properly then rerun the libmesh build script and try to compile MOOSE again."
However, I appear to have the proper PETSC and LIBMESH path in my environment.
"echo$PETSC_DIR "returns "spack/opt/spack/linux-fedora33-skylake/gcc-8.4.0/petsc-3.14.1-k3f2iiz7zaifdrdr5bdjwih6nun3lxn3"
Which is the proper path on my machine.
Any ideas?

[fdkong]

Did you run ./scripts/update_and_rebuild_libmesh.sh already to build libmesh? How did you build PETSc (using ./scripts/update_and_rebuild_petsc.sh)?

[dorton21]

I haven't run either. I'm attempting to set up Moose to run on the package manager Spack (https://spack.io/). So it's been a very, shall we say, custom installation process. Attempting to run the ./scripts/update_and_rebuild_libmesh.sh just now returns an error regarding no file or directory. I believe it is looking under the submodule libmesh. However, running "echo$LIBMESH_DIR" reveals the proper file path.

[fdkong]

Did you follow these instructions carefully? https://mooseframework.inl.gov/getting_started/installation/offline_installation.html

[dorton21]

More or less. Spack takes the moose tarball and extracts it to a directory. Then inside of the package for spack, we specify dependencies such as libmesh and petsc. Spack will then extract and install libmesh and petsc to the same directory moose is installed. Inside of the petsc and libmesh dependencies are specified and also installed in the same directory as everything else. The PATHS are automatically specified. The only step that I've struggled to incorporate is the update_and_rebuild. Hopefully, this makes sense... I should mention the goal is to make this an automatic process so all a user has to do is "spack install moose" and then all the user needs to do is run ./PATH/TO/MOOSE/scripts/stork.sh AppName

[loganharbour]

Hmm. It seems like you're trying to setup a development environment for people on your HPC.

In a HPC environment, I'm much more a fan of giving users the base envs (mpich, cmake, python, more or less) and having them build PETSc and libMesh. Reason being, with the continuous changes that come with MOOSE, we update libMesh and PETSc in a regular manner. The preferred practice then is for users to manage their apps on their own with a moose submodule, which could vastly change which versions of PETSc and libMesh they need.

An alternative would be to provide PETSc, $PETSC_DIR, and $PETSC_ARCH, which changes far less often than libMesh, and then have them build their own libMesh.

This is just my $0.02. I can see some validity to actually providing a compiled MOOSE for users and having them update it as your admins update it. Internally, we use spack and modules for all of our build machines. For INL's HPC services, we provide the base compiler spack and PETSc (although we're considering transitioning away from that) via spack and modules and have users (developers) compiler libMesh.

[loganharbour]

Can you share the output of of the configure call for libMesh?

[dorton21]

INFO: LIBMESH_DIR set - overriding default installed path
INFO: No cleaning will be done in specified path
./scripts/update_and_rebuild_libmesh.sh: line 151: /home/las_djorton/spack/opt/spack/linux-fedora33-skylake/gcc-8.4.0/moose-20-10-27-fch3jy6pi5r5htldeik5pevjzuzqoizt/scripts/../libmesh/configure: No such file or directory
make: *** No targets specified and no makefile found. Stop.

[fdkong]

spack is able to have set LIBMESH_DIR as it does for PETSC_DIR? If you have libmesh and PETSc installed correctly via spack, then you do NOT need to run update_and_rebuild_libmesh/petsc.sh.

I think the problem right now is that spack did not set LIBMESH_DIR, and MOOSE can not figure out where libmesh is.

Set LIBMESH_DIR to point to libmesh, and then recompile moose again

[dorton21]

@fdkong I do appreciate your tip, however, I've already attempted this and it has not solved the problem. I do like that you confirmed my original thought (that the setup steps are redundant in spack) however I still have the issue of it not finding the correct PATH. However, when I check the PATH that is listed (I run the cmd $LIBMESH_DIR and $PETSC_DIR to check the path) the PATH is correct. Thus my confusion. Additionally, I would like to mention I have the PETSC and LIBMESH modules loaded. Still I get the error:
"""/home/spack/opt/spack/linux-fedora33-skylake/gcc-8.4.0/moose-20-10-27-ithupfgwabwausyf3ygn7lutwbskvd73/framework/build/header_symlinks/Moose.h:235:2: error: #error PETSc has not been detected, please ensure your environment is set up properly then rerun the libmesh build script and try to compile MOOSE again.
#error PETSc has not been detected, please ensure your environment is set up properly then rerun the libmesh build script and try to compile MOOSE again.
"""
"""
/home/spack/opt/spack/linux-fedora33-skylake/gcc-8.4.0/moose-20-10-27-ithupfgwabwausyf3ygn7lutwbskvd73/framework/build/header_symlinks/DualReal.h:17:10: fatal error: metaphysicl/metaphysicl_version.h: No such file or directory
#include "metaphysicl/metaphysicl_version.h"
"""

[dorton21]

I figured out the reason this wasn't working was LIbmesh was not finding petsc but the moose package was. I had to specify within my libmesh install the path to petsc.