Skip to content

Westley

Jump to bottom
Glen Hansen edited this page Feb 11, 2019 · 6 revisions

Set up your build environment

This build is based on the 2016 Intel compilers

	. /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64
	export PATH=/usr/local/trilinos/MPI_REL/bin:/usr/local/gcc-5.1.0/bin:/opt/intel/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:$HOME/bin
	. /opt/intel/impi/5.1.2.150/bin64/mpivars.sh
	export LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.3.187/compiler/lib/mic:/usr/local/gcc-5.1.0/lib64:$LD_LIBRARY_PATH
	export I_MPI_MIC=1

These install directions assume that a base install directory of /usr/local/mic is used

Install the required Third Party Libraries (TPLs) for Trilinos

Build HWLOC

  • HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

  • Version 1.10

      cd /usr/local/src
  wget http://www.open-mpi.org/software/hwloc/v1.10/downloads/hwloc-1.10.0.tar.bz2
  tar -xvf hwloc-1.10.0.tar.bz2
  cd hwloc-1.10.0
  ./configure CC=mpiicc CFLAGS="-O3 -mmic -fopenmp" CXX=mpiicpc \
  	CXXFLAGS="-O3 -mmic -fopenmp" LDFLAGS=-static \
  	--host=x86_64-k1om-linux --enable-shared=no \
  	--enable-static=yes --prefix=/usr/local/mic
  make -j 8
  make install

Building ZLib

  • Version 1.2.8

      cd /usr/local/src
  wget http://zlib.net/zlib-1.2.8.tar.gz
  tar -xvf zlib-1.2.8.tar.gz
  cd zlib-1.2.8
  CC=mpiicc CXX=mpiicpc CFLAGS="-O3 -mmic -fopenmp" CXXFLAGS="-O3 -mmic -fopenmp" \
  	./configure --64 --static --prefix=/usr/local/mic
  make install

Building HDF5

  • Version 1.8.17

Cross compiling HDF5 for the MIC can be a challenge. Please see the instructions at:

	http://linux4hippos.blogspot.com/2013/05/cross-compile-hdf5-for-intel-xeon-phi.html

Build Instructions: Note

  • The above website is based on HDF5-1.8.9 but they work with version HDF5-1.8.17

  • The above website, step (1), instructs one to set TEST_SCRIPT = in several files. These instructions were followed here.

  • as_fn_exit was commented out in three places in the configure script as described in step (2) above

      cd hdf5-1.8.17
  ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp" \
  	CFLAGS="-O3 -mmic -fopenmp" FCFLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux \
  	--enable-parallel --enable-shared=no --with-zlib=/usr/local/mic --prefix=/usr/local/mic
  make -j 8

Note that the build fails as expected (above website, step (6)). Follow the instructions given to proceed.

Building parallel-netcdf

  • Version 1.7.0

  • Configure and build:

      ./configure --prefix=/usr/local/mic MPICC=mpiicc MPIF77=mpiifort MPIF90=mpiifort MPICXX=mpiicpc \
  CXXFLAGS="-O3 -mmic -fopenmp" CFLAGS="-O3 -mmic -fopenmp" \
  F77FLAGS="-O3 -mmic -fopenmp" F90FLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux
  make -j 8 
  make install

Building Netcdf

  • Version 4.4.0

  • Edit the file include/netcdf.h. Make the following edits near line 265:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
  #define NC_MAX_ATTRS    8192    
  #define NC_MAX_VARS   524288    /* max variables per file */
  #define NC_MAX_NAME   256 
  #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */

  • Now, configure using CMake and build:

      cd hdf5-1.8.17
  ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
  	CFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
      LDFLAGS="-O3 -mmic -fopenmp -L/usr/local/mic/lib" \
      FCFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" -host=x86_64-k1om-linux \
  	--prefix=/usr/local/mic --enable-netcdf-4 --disable-dap --enable-shared=no \
      --disable-utilities --disable-testsets --enable-pnetcdf
  make -j 8
  make install

Building boost

  • Version 1.58.0

      cd /usr/local/src/
  wget http://sourceforge.net/projects/boost/files/boost/1.58.0/boost_1_58_0.tar.gz/download boost_1_58_0.tar.gz
  tar -xvf boost_1_58_0.tar.gz
  cd boost_1_58_0

  • Build boost with the commands

      echo "using intel : mic : : <cxxflags>\"-O3 -mmic -fopenmp\" ;" \
  	>> ~/user-config.jam
  echo "using mpi : /opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpiicpc : <cxxflags>\"-I/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include -mmic -fopenmp\" ;" \
  	>> ~/user-config.jam
  ./bootstrap.sh --with-toolset=intel \
  --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
  --prefix=/usr/local/mic
  ./b2 -j 8
  ./b2 -j 8 install

  • The above bjam command gave me an error, so I removed the following lines from project-config.jam:

      if ! intel-darwin in [ feature.values <toolset> ]
  {
      using intel-darwin ;
  }

Building ParMetis

  • Version 4.0.3

      cd /usr/local/src/
  wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
  tar -xvf parmetis-4.0.3.tar.gz
  cd parmetis-4.0.3

  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
  vi metis.h
  #define IDXTYPEWIDTH 64
  #define REALTYPEWIDTH 64

  • Run CMake to build parmetis

      cd /usr/local/src/parmetis-4.0.3
  cd build
  export metis=/usr/local/src/parmetis-4.0.3/metis

  cmake \
  -DCMAKE_INSTALL_PREFIX=/usr/local/mic \
  -DMETIS_PATH=$metis \
  -DGKLIB_PATH=$metis/GKlib \
  -DCMAKE_C_COMPILER=mpiicc \
  -DCMAKE_C_FLAGS="-O3 -mmic -fopenmp" \
  -DCMAKE_CXX_COMPILER=mpiicpc \
  -DCMAKE_CXX_FLAGS="-O3 -mmic -fopenmp" \
  ..

  make -j 8
  make install

  cp libmetis/libmetis.a /usr/local/mic/lib
  cp $metis/include/metis.h /usr/local/mic/include

Build Trilinos

  • Download Trilinos

      cd /usr/local/src
  git clone [email protected]:trilinos/Trilinos.git

  • Download SCOREC

      cd /usr/local/src/Trilinos
  git clone [email protected]:SCOREC/core.git SCOREC

  • Configure the Trilinos build

      mkdir build
  cd build
  ./do-configure

  • Example do-configure script executed above

      #!/bin/sh
  #
  
  export TRILINOS_HOME=/usr/local/src/Trilinos
  BUILD_DIR=`pwd`
  INSTALL_DIR=/usr/local/mic/trilinos/MPI_MIC
  BOOST_DIR=/usr/local/mic/boost-1.58.0
  MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64
  NETCDF=/usr/local/mic
  HDFDIR=/usr/local/mic
  ZLIB_DIR=/usr/local/mic
  PARMETISDIR=/usr/local/mic
  HWLOC_PATH=/usr/local/mic
  INTEL_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/mic
  
  cmake \
      -Wno-dev \
      -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
      -D Trilinos_ENABLE_SCOREC:BOOL=ON \
      -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
      -D CMAKE_BUILD_TYPE:STRING=RELEASE \
      -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
      -D BUILD_SHARED_LIBS:BOOL=OFF \
      -D TPL_ENABLE_MPI:BOOL=ON \
      -D MPI_BASE_DIR:PATH="$MPI_BASE_DIR"\
      -D CMAKE_CXX_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicpc" \
      -D CMAKE_C_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicc" \
      -D CMAKE_Fortran_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiifort" \
      -D CMAKE_AR:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xiar \
      -D CMAKE_LINKER:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xild \
      -D CMAKE_CXX_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
      -D CMAKE_C_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
      -D CMAKE_Fortran_FLAGS:STRING="-O3 -mmic -DPREC_TIMER -fopenmp" \
      -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
      -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
      -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
      -D Trilinos_ENABLE_Teuchos:BOOL=ON \
     \
      -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
      -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
      -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
     \
      -D Trilinos_ENABLE_SEACASNemesis:BOOL=ON \
      -D Trilinos_ENABLE_Zoltan:BOOL=ON \
      -D Zoltan_ENABLE_METIS:BOOL=OFF \
      -D Trilinos_ENABLE_Kokkos:BOOL=ON \
      -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
      -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
      -D Trilinos_ENABLE_KokkosExample:BOOL=OFF \
      -D Trilinos_EXTRA_LINK_FLAGS:STRING="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
     \
      -D Kokkos_ENABLE_Serial:BOOL=OFF \
      -D Kokkos_ENABLE_OpenMP:BOOL=ON \
      -D Kokkos_ENABLE_Pthread:BOOL=OFF \
      -D Kokkos_ENABLE_Cuda:BOOL=OFF \
      -D Trilinos_ENABLE_OpenMP:BOOL=ON \
      -D TPL_ENABLE_CUDA:BOOL=OFF \
      -D TPL_ENABLE_CUSPARSE:BOOL=OFF \
  \
      -D TPL_ENABLE_Pthread:BOOL=OFF \
      -D TPL_ENABLE_HWLOC:BOOL=OFF \
  \
      -D TPL_ENABLE_Boost:BOOL=ON \
      -D TPL_ENABLE_BoostLib:BOOL=ON \
      -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
      -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
      -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
  \
      -D TPL_ENABLE_Netcdf:STRING=ON \
      -D TPL_Netcdf_PARALLEL:BOOL=ON \
      -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
      -D TPL_Netcdf_LIBRARIES:FILEPATH="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
      -D TPL_ENABLE_Pnetcdf:STRING=ON \
      -D Pnetcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
      -D Pnetcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
  \
      -D TPL_ENABLE_HDF5:STRING=ON \
      -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
      -D HDF5_LIBRARY_DIRS:PATH="$HDFDIR/lib;$ZLIB_DIR/lib" \
  \
      -D TPL_ENABLE_Zlib:STRING=ON \
      -D Zlib_INCLUDE_DIRS:PATH=$ZLIB_DIR/include \
      -D Zlib_LIBRARY_DIRS:PATH=$ZLIB_DIR/lib \
  \
      -D TPL_ENABLE_BLAS:BOOL=ON \
      -D TPL_ENABLE_LAPACK:BOOL=ON \
      -D BLAS_LIBRARY_DIRS:FILEPATH=${INTEL_DIR} \
      -D TPL_BLAS_LIBRARIES:STRING="${INTEL_DIR}/libmkl_intel_lp64.a;${INTEL_DIR}/libmkl_sequential.a;${INTEL_DIR}/libmkl_core.a" \
      -D LAPACK_LIBRARY_NAMES:STRING= \
  \
      -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
      -D TPL_ENABLE_X11=OFF \
      -D FC_FN_CASE=LOWER \
      -D FC_FN_UNDERSCORE=UNDER  \
      -D Trilinos_ENABLE_MueLu:BOOL=OFF \
  	-D TPL_FIND_SHARED_LIBS:BOOL=OFF \
  \
  $EXTRA_ARGS \
  ${TRILINOS_HOME}
  
  exit 0

  • Build Trilinos

      make -j 8
  make install

Build Albany

  • Clone the repo

      cd /usr/local/src
  git clone [email protected]:SNLComputation/Albany.git
  cd Albany

  • Configure the Albany build

      mkdir build
  cd build
  ./do-configure

  • CMake configure script ./do-configure:

      #!/bin/sh
  
  export ALB_64BIT_INT=OFF
  export ALB_EPETRA=ON
  export ALB_SEE=OFF
  export ALB_ENABLE_SCOREC=ON
  export ALB_ENABLE_LCM=ON
  export ALB_ENABLE_LAME=ON
  export ALB_ENABLE_HYDRIDE=OFF
  export ALB_ENABLE_QCAD=OFF
  export ALB_ENABLE_MOR=OFF
  export ALB_ENABLE_ASCR=OFF
  export ALB_ENABLE_AERAS=OFF
  export ALB_ENABLE_FELIX=OFF
  export ALB_ENABLE_AlbanyCI=OFF
  export ALB_DEBUG=OFF
  export ALB_ENABLE_CHECK_FPE=OFF
  export ALB_ENABLE_SPECULATIVE=OFF
  export ALB_ENABLE_SG_MP=OFF
  export ALB_ENABLE_PERF_TESTS=ON
  export TRILINOS_INSTALL_DIR=/usr/local/host/trilinos/MPI_MIC
  export ALB_VERBOSE=OFF
  
  cmake \
        -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
        -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
        -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
        -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
        -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
        -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
        -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \
        -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
        -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
        -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
        -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
        -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
        -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
        -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
        -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
        -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
        -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
        -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
        -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
        -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
        -D ENABLE_SEE:BOOL=${ALB_SEE} \
        -D ALBANY_CXX_FLAGS:STRING=${ALB_FLAGS} \
  	  -D ENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
  	  -D ENABLE_CROSS_COMPILE:BOOL=ON \
  	  -D ALBANY_MPI_OPTIONS:BOOL=ON \
  	  -D ALBANY_MPI_EXEC:STRING=${MPI_BASE_DIR}/bin/mpiexec.hydra \
  	  -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING=-n \
  	  -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING=4 \
  	  -D ALBANY_MPI_TRAILING_OPTIONS:STRING='-hosts mic1 -ppn 4 -env OMP_NUM_THREADS 56 -env KMP_AFFINITY balanced -binding domain=omp -env LD_LIBRARY_PATH ${INTEL_DIR}' \
  \
  ..

  • Build Albany

      make -j 8
  make install

  • Test Albany

      ctest

Docker Image

SPACK Package

ToDo List

Albany FAQs

Capabilities

Sub-Projects

Compiling

Nightly Tests

Code HowTo

User Group Meetings

PAALS Tutorials

Wiki HowTo

Visualization Utilities

Good Commits

Python and Albany

Clone this wiki locally