Building MOOSE without Nvidia Library

[moravveji]

Dear MOOSE maintainers

I am trying to build MOOSE on our Tier-1 machine (Intel Broadwell and SkyLake) for a group of users, using foss/2018a toolchain. There are no nvidia libraries available on the compute nodes, since there need be no graphics there. However, the nvidia libraries are available on the login nodes. Also, the architecture of our login nodes and compute nodes are different, hence MOOSE must be built on the compute nodes.

Now, here is the situation:

• it is possible to build MOOSE on the login nodes (having nvidia libs) and run them on the broadwell nodes; but such binaries run suboptimally on Skylake nodes. Hence, it seems that the nvidia libraries are not used at runtime, and it is possible to ignore them

• it is not possible to compile MOOSE on the compute nodes, due to the error below:

  CXX      apps/version.o
  CXX      utilities/src/metaphysicl_version.lo
  CXXLD    libmetaphysicl.la
/dirApps/binutils/2.28-GCCcore-6.4.0/bin/ld.gold: error: cannot f
ind -lnvidia-ml
collect2: error: ld returned 1 exit status
make[4]: *** [libmetaphysicl.la] Error 1
make[4]: Leaving directory `/myData/work/MOOSE/moo
se/libmesh/build/contrib/metaphysicl/src'
make[3]: *** [all] Error 2
make[3]: Leaving directory `/myData/work/MOOSE/moo
se/libmesh/build/contrib/metaphysicl/src'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/myData/work/MOOSE/moo
se/libmesh/build/contrib/metaphysicl'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/myData/work/MOOSE/moo
se/libmesh/build/contrib'
make: *** [all-recursive] Error 1

Now my questions:

• How can I get around the nvidia dependency?
• How do I (re)set the LDFLAGS and LIBS env. vars. when executing the update_and_rebuild_libmesh.sh script?
• I also would like to add -march=native flag too, if not already introduced by you
• Do you need more inputs from my side?

I have attached the full logs and the bash script which I use building MOOSE.
I thank you in advance for your support.

Regards
Ehsan
moose.log.tar.gz

[cticenhour]

Hi @moravveji,
Thank you for your patience as we get through a backlog of issues, tickets, and support requests! I have moved your question to our GitHub Discussions forum, where we prefer users post questions and support requests before the creation of issues/bug reports. Because your question is related to build dependencies and platform-specific questions, I am tagging @milljm to provide some input on your problem. He is the MOOSE team member in charge of our build system. Let me know if you have more questions!
Thanks,
Casey

[moravveji]

Thanks Casey. I stay tuned.

[cticenhour]

Since this has sat a while, I'll take a crack at answering some of your questions.

• I am not sure what nVidia dependency you are referring to - MOOSE doesn't strictly require particular libraries and architectures. You may be seeing a check for OpenCL that MetaPhysicL performs....in which case (if you don't need openCL support), I would try adding --disable-opencl to the update_and_rebuild_libmesh.sh script when libMesh is configured (look at lines 151 - 162 in the current version of the script)

• LDFLAGS and LIBS should be able to be set directly on the command line. (export LDFLAGS=[flags] and export LIBS=[flags]). LibMesh configure should pick them up. @roystgnr can comment more on this if I am wrong.

• Adding --enable-march to the configure lines in the libmesh script (as mentioned above) allows libmesh to determine the correct -march flag for the system, if I'm interpreting libmesh configure correctly. Try this first. If this doesn't work, you should be able to add -march=native to CFLAGS and/or CXXFLAGS for your system.

Hopefully this helps!

[moravveji]

Thanks a lot @cticenhour for the tips. I just incorporated them all, and MOOSE builds on our SkyLake machines like a charm. No more missing Nvidia library fuss ...
Next step is testing the build and performance by our users.

By the way, the two very attractive options that you mentioned (--enable-march and --disable-opencl) were not immediately fetchable by myself. Can you please point me out to the inventory of options that I can still use? Maybe I can still optimize the build further.

[cticenhour]

Glad it's working well! Most of these options can be found by navigating to your libmesh directory, and performing the command ./configure -h in your Terminal. Using this list, looks like you have access to --with-gcc-arch=<arch> in order to set a particular march configuration instead of guessing, which may give you some more optimization. I don't see --disable-opencl in this list, but it is a MetaPhysicL configure option, so perhaps it's being passed through by the libmesh configure script (not sure). I found that option by searching in the MetaPhysicL repository, located at (https://github.com/roystgnr/MetaPhysicL). Specifically the ax_check_cl.m4 file used as a configure file for this check in that code.

[roystgnr]

I don't see --disable-opencl in this list, but it is a MetaPhysicL configure option, so perhaps it's being passed through by the libmesh configure script (not sure).

That's correct. MetaPhysicL allows an optional build with VexCL, and tests for OpenCL to see if this is possible. libMesh doesn't have anything to do (directly) with either, but does pass any configure arguments along to its subpackages.

Trying to configure on a different system than you build on is pretty much never going to be supported, I'm afraid - we'd have to turn off all our autodetection to make that safe. And that "-lnvidia-ml" must getting inserted by a library on an Nvidia-possessing system here; ax_check_cl.m4 only tries out -lOpenCL, -lCL, and -lclparser, not any library with a vendor-specific name.

[moravveji]

@roystgnr : and still, somehow magically, the unwanted libnvidia-ml is asked for, as I had demonstrated in the original ticket.
But, with --disable-opencl life is happy again.
So, thank you guys.

[moravveji]

This ticket maybe closed now.

[cticenhour]

@moravveji This new discussion format never truly closes like a regular GitHub issue, since it is more like a forum. I selected my answer above as the answer that solved your issue, and that is all we need to do.