-
Notifications
You must be signed in to change notification settings - Fork 89
Rhel6 using Intel
Glen Hansen edited this page Apr 18, 2017
·
9 revisions
This set of build instructions are for the Albany suite on Rhel6 linux.
The compilers are Intel 17.0, Intel MPI 5.1, and RHEL 6 linux.
Modules on CEE:
module load sierra-devel/intel-17.0.1-intelmpi-5.1
module load sierra-mkl/17.0-2017.2.174
Trilinos requires a set of "third party libraries" (TPLs) to support Albany. The actual libraries needed depends on the final Albany configuration desired. These instructions build a superset, all the libraries needed to support any Albany configuration.
Notes: In this example
- The TPLs are installed in
/projects/albany
-
Version 1.2.11
CC=mpiicc CFLAGS=-O1 ./configure --64 --prefix=/projects/albany/intel5.1 make -j 4 make install
-
git clone [email protected]:jbeder/yaml-cpp.git
INTEL_MPI_DIR=/sierra/sntools/SDK/compilers/intel/composer_xe_2017.2.174/compilers_and_libraries_2017.2.174/linux/mpi/intel64 cmake \ -DCMAKE_CXX_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicpc \ -DCMAKE_CXX_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \ -DCMAKE_C_COMPILER:STRING=${INTEL_MPI_DIR}/bin/mpiicc \ -DCMAKE_C_FLAGS:STRING='-O3 -DNDEBUG -diag-disable=cpu-dispatch -mkl=sequential' \ -DCMAKE_INSTALL_PREFIX:PATH=/projects/albany/intel5.1 \ -DCMAKE_BUILD_TYPE:STRING=Release \ -DBUILD_SHARED_LIBS:BOOL=ON \ -DYAML_CPP_BUILD_TOOLS:BOOL=OFF \ .. make -j 4 make install
-
Version 1.8.17
./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" \ CFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" \ FCFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" --enable-parallel \ --with-zlib=/projects/albany/intel5.0 --prefix=/projects/albany/intel5.0 make -j 4 make install
-
Version 1.7.0
-
Configure and build:
./configure --prefix=/projects/albany/intel5.0 CC=mpiicc F77=mpiifort F90=mpiifort CXX=mpiicpc \ CXXFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" CFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" \ FFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" FCFLAGS="-xHOST -O3 -no-prec-div -fp-model fast=2 -fPIC" \ --with-mpi=/sierra/sntools/SDK/mpi/intel/5.0 make -j 8 make install
-
Version 4.4.0
-
Edit the file
include/netcdf.h. Make the following edits near line 265:#define NC_MAX_DIMS 65536 /* max dimensions per file */ #define NC_MAX_ATTRS 8192 #define NC_MAX_VARS 524288 /* max variables per file */ #define NC_MAX_NAME 256 #define NC_MAX_VAR_DIMS 8 /* max per variable dimensions */ -
Now, configure using CMake and build:
./configure CC=mpiicc FC=mpiifort CXX=mpiicpc \ CXXFLAGS="-fPIC -I/projects/albany/intel5.0/include -xHOST -O3 -no-prec-div -fp-model fast=2" \ CFLAGS="-fPIC -I/projects/albany/intel5.0/include -xHOST -O3 -no-prec-div -fp-model fast=2" \ LDFLAGS="-fPIC -L/projects/albany/intel5.0/lib -xHOST -O3 -no-prec-div -fp-model fast=2" \ FCFLAGS="-fPIC -I/projects/albany/intel5.0/include -xHOST -O3 -no-prec-div -fp-model fast=2" \ --prefix=/projects/albany/intel5.0 --enable-netcdf-4 --enable-pnetcdf make -j 8 make install
-
Version 1.55.0
-
Build boost with the commands
echo "using intel-linux : : /sierra/sntools/SDK/compilers/intel/composer_xe_2015.3.187/bin/intel64/icpc : <cxxflags>"-xHOST -O3 -no-prec-div -fp-model fast=2" ;" >> ./tools/build/v2/user-config.jam echo "using mpi : /sierra/sntools/SDK/mpi/intel/5.0.1.035/bin64/mpiicpc : <cxxflags>"-I/sierra/sntools/SDK/mpi/intel/5.0.1.035/intel64/include -xHOST -O3 -no-prec-div -fp-model fast=2" ;" >> ./tools/build/v2/user-config.jam ./bootstrap.sh --with-toolset=intel-linux --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception --prefix=/projects/albany/intel5.0 ./b2 toolset=intel-linux -j 4 ./b2 toolset=intel-linux -j 4 install -
The above bjam command gave me an error, so I removed the following lines from project-config.jam:
if ! intel-linux in [ feature.values <toolset> ] { using intel-linux ; }
-
Version 4.0.3
wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz tar -xvf parmetis-4.0.3.tar.gz -
Edit
metis.hto use anIDXTYPEWIDTHof 64cd parmetis-4.0.3/metis/include vi metis.h #define IDXTYPEWIDTH 64 #define REALTYPEWIDTH 64 -
Run CMake to build
parmetiscd /projects/albany/src/parmetis-4.0.3 cd build export metis=/projects/albany/src/parmetis-4.0.3/metis/ cmake \ -DCMAKE_INSTALL_PREFIX=/projects/albany/intel5.0 \ -DMETIS_PATH=$metis \ -DGKLIB_PATH=$metis/GKlib \ -DCMAKE_C_COMPILER=mpiicc \ -DCMAKE_CXX_COMPILER=mpiicpc \ -DCMAKE_C_FLAGS=-xHOST -O3 -no-prec-div -fp-model fast=2 \ -DCMAKE_CXX_FLAGS=-xHOST -O3 -no-prec-div -fp-model fast=2 \ .. make -j 4 make install cp libmetis/libmetis.a /projects/albany/intel5.0/parmetis-4.0.3/lib cp $metis/include/metis.h /projects/albany/intel5.0/parmetis-4.0.3/include
-
Version 4.3
-
Build paths
mkdir /projects/albany/intel5.0/SuperLU_4.3 mkdir /projects/albany/intel5.0/SuperLU_4.3/include mkdir /projects/albany/intel5.0/SuperLU_4.3/lib -
Edit
make.incfor your machine and environmentPLAT = _linux SuperLUroot = /projects/albany/intel5.0/SuperLU_4.3 SUPERLULIB = $(SuperLUroot)/lib/libsuperlu_4.3.a TMGLIB = libtmglib.a MKLHOME = /sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210 BLASDEF = -DUSE_VENDOR_BLAS BLASLIB = -L${MKLHOME}/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread LIBS = $(SUPERLULIB) $(BLASLIB) ARCH = xiar ARCHFLAGS = cr RANLIB = ranlib CC = mpiicc CFLAGS = -O3 NOOPTS = FORTRAN = mpiifort FFLAGS = -O3 LOADER = $(CC) LOADOPTS = CDEFS = -DAdd_ MATLAB = /usr/sww/matlab -
Build SuperLU
make cd SRC cp *.h /projects/albany/intel5.0/SuperLU_4.3/include -
Note: Do not make SuperLU in parallel (-j * make argument) as the build will be incomplete.
-
HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.
-
Version 1.8
./configure CC=mpicc CXX=mpiicpc --prefix=/projects/albany/intel5.0 make -j 4 make install
Notes: In this example
-
The Trilinos source is installed in
/home/gahanse/Codes/Trilinos, using a command likecd /home/gahanse/Codes git clone [email protected]:trilinos/Trilinos.git -
The SCOREC tools are installed as a subdirectory of Trilinos, and are build automatically by the Trilinos build process. The commands to do this are
cd /home/gahanse/Codes/Trilinos git clone https://github.com/SCOREC/core.git SCOREC -
do-configure script
#!/bin/sh # export TRILINOS_HOME=/projects/albany/src/Trilinos BUILD_DIR=`pwd` INSTALL_DIR=/projects/albany/trilinos/MPI_INTEL BOOST_DIR=/projects/albany/intel5.0/boost-1.58.0 MPI_BASE_DIR=/sierra/sntools/SDK/mpi/intel/5.0.1.035 NETCDF=/projects/albany/intel5.0 HDFDIR=/projects/albany/intel5.0 ZLIB_DIR=/projects/albany/intel5.0 PARMETISDIR=/projects/albany/intel5.0 INTEL_DIR=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210/mkl/lib/intel64 MKL_PATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2016.3.210 HWLOC_PATH=/projects/albany LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/libmkl_core.a $MKL_PATH/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group" cmake \ -Wno-dev \ -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \ -D Trilinos_ENABLE_SCOREC:BOOL=ON \ -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \ -D CMAKE_BUILD_TYPE:STRING=NONE \ -D CMAKE_CXX_COMPILER:FILEPATH=$MPI_BASE_DIR/bin64/mpiicpc \ -D CMAKE_C_COMPILER:FILEPATH=$MPI_BASE_DIR/bin64/mpiicc \ -D CMAKE_Fortran_COMPILER:FILEPATH=$MPI_BASE_DIR/bin64/mpiifort \ -D CMAKE_CXX_FLAGS:STRING="-xHOST -O3 -no-prec-div -fp-model fast=2 -mkl=sequential -DNDEBUG" \ -D CMAKE_C_FLAGS:STRING="-xHOST -O3 -no-prec-div -fp-model fast=2 -mkl=sequential -DNDEBUG" \ -D CMAKE_Fortran_FLAGS:STRING="-xHOST -O3 -no-prec-div -fp-model fast=2 -mkl=sequential -DNDEBUG" \ -D CMAKE_AR:FILEPATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2015.2.164/bin/intel64/xiar \ -D CMAKE_LINKER:FILEPATH=/sierra/sntools/SDK/compilers/intel/composer_xe_2015.2.164/bin/intel64/xild \ -D Rythmos_ENABLE_DEBUG:BOOL=ON \ -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \ -D Tpetra_INST_INT_LONG_LONG:BOOL=ON \ -D Tpetra_INST_INT_INT:BOOL=ON \ -D Tpetra_INST_DOUBLE:BOOL=ON \ -D Tpetra_INST_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \ -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \ -D Tpetra_INST_INT_LONG:BOOL=OFF \ -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \ -D KokkosClassic_DefaultNode:STRING="Kokkos::Compat::KokkosSerialWrapperNode" \ -D Trilinos_ENABLE_Kokkos:BOOL=ON \ -D Trilinos_ENABLE_KokkosCore:BOOL=ON \ -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="SERIAL" \ -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \ -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \ -D Kokkos_ENABLE_Serial:BOOL=ON \ -D Kokkos_ENABLE_OpenMP:BOOL=OFF \ -D Kokkos_ENABLE_Pthread:BOOL=OFF \ \ -D TPL_ENABLE_MPI:BOOL=ON \ -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \ -D MPI_EXEC:FILEPATH=$MPI_BASE_DIR/bin64/mpiexec.hydra \ \ -D TPL_ENABLE_Pthread:BOOL=OFF \ \ -D TPL_ENABLE_Boost:BOOL=ON \ -D TPL_ENABLE_BoostLib:BOOL=ON \ -D TPL_ENABLE_BoostAlbLib:BOOL=ON \ -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \ -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \ \ -D TPL_ENABLE_Netcdf:STRING=ON \ -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \ -D TPL_Netcdf_LIBRARIES:FILEPATH="${NETCDF}/lib64/libnetcdf.a;${HDFDIR}/lib/libhdf5_hl.a;${HDFDIR}/lib/libhdf5.a;${ZLIB_DIR}/lib/libz.a" \ \ -D TPL_ENABLE_HDF5:STRING=ON \ -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D HDF5_LIBRARY_NAMES:STRING="" \ \ -D TPL_ENABLE_Zlib:STRING=ON \ -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \ -D Zlib_LIBRARY_NAMES:STRING="" \ \ -D TPL_ENABLE_BLAS:STRING=ON \ -D TPL_ENABLE_LAPACK:STRING=ON \ -D TPL_BLAS_LIBRARIES:STRING="${INTEL_DIR}/libmkl_intel_lp64.a;${INTEL_DIR}/libmkl_sequential.a;${INTEL_DIR}/libmkl_core.a" \ -D LAPACK_LIBRARY_NAMES:STRING="" \ \ -D TPL_ENABLE_ParMETIS:STRING=ON \ -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \ -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \ \ -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \ \ -D Trilinos_ENABLE_Moertel:BOOL=ON \ -D Moertel_ENABLE_EXAMPLES:BOOL=ON \ -D Moertel_ENABLE_Morkon:BOOL=ON \ -D Trilinos_ENABLE_TriKota:BOOL=OFF \ -D HAVE_INTREPID_KOKKOSCORE:BOOL=ON \ -D Trilinos_ENABLE_SECONDARY_TESTED_CODE=OFF \ -D Trilinos_ENABLE_Amesos2:BOOL=ON \ -D Amesos2_ENABLE_KLU2:BOOL=ON \ -D TPL_FIND_SHARED_LIBS:BOOL=OFF \ \ $EXTRA_ARGS \ ${TRILINOS_HOME}
To build Trilinos, use an out-of-source build. I usually make a build directory that matches the configuration of Trilinos that I am building in the Trilinos root directory:
cd /home/gahanse/Codes/Trilinos
mkdir build
cd build
mkdir MPI_REL
-
Place the above do-configure file into the MPI_REL directory, change directories into MPI_REL, and perform the configure, build, and install:
cd MPI_REL ./do-configure make -j 4 make -j 4 install
-
CMake configure script:
#!/bin/sh export ALB_64BIT_INT=OFF export ALB_EPETRA=ON export ALB_ENABLE_SCOREC=ON export ALB_ENABLE_LCM=ON export ALB_ENABLE_LAME=OFF export ALB_ENABLE_HYDRIDE=OFF export ALB_ENABLE_QCAD=OFF export ALB_ENABLE_MOR=OFF export ALB_ENABLE_ASCR=OFF export ALB_ENABLE_AERAS=ON export ALB_ENABLE_FELIX=ON export ALB_ENABLE_AlbanyCI=OFF export ALB_DEBUG=OFF export ALB_ENABLE_CHECK_FPE=OFF export ALB_ENABLE_SPECULATIVE=OFF export ALB_ENABLE_SG_MP=OFF export ALB_ENABLE_PERF_TESTS=ON export TRILINOS_INSTALL_DIR=/projects/albany/trilinos/MPI_INTEL export ALB_VERBOSE=OFF cmake \ -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \ -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \ -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \ -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \ -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \ -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \ -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \ -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \ -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \ -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \ -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \ -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \ -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \ -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \ -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \ -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \ -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \ -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \ -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \ -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \ \ ..
Clone the Albany repo in a suitable place:
cd /home/ghansen/Codes
git clone https://github.com/gahansen/Albany.git
-
Create an out-of-source build directory
cd Albany mkdir build -
Place the above configure script into the build directory, then configure, build, and test
cd build ./buildAlbany make -j 4 ctest