-
Notifications
You must be signed in to change notification settings - Fork 2
Compiling on LC Machines
The steps below can be used to compile pylibROM on LC machines.
Note: These steps assume that libROM has already been compiled on LC following the steps for compiling libROM on LC machines.
It is recommended to use a virtualenv to setup pylibROM on LC machines. These steps will create a virtualenv in your home directory with a folder based on system type: $HOME/local/$SYS_TYPE.
Note: PyMFEM needs an older version of pip and setuptools to install correctly. The virtualenv below will be created with the necessary versions.
module load python/3.9.12
virtualenv --system-site-packages --pip 22.0.4 --setuptools 58.1.0 --wheel 0.37.0 --download $HOME/local/$SYS_TYPE
source $HOME/local/$SYS_TYPE/bin/activate
export PYTHONUSERBASE=$HOME/local/$SYS_TYPE
Ensure that the correct python is being found with which python and that the site packages are set to the correct $HOME/local/$SYS_TYPE path using python -m site --user-site
A newer swig is needed for PyMFEM since the system default version is too old.
pip install swig==4.1.1
Note: After the virtualenv is created, these steps do not need to be repeated. Only activating the virtualenv and setting the site-paths are needed:
source $HOME/local/$SYS_TYPE/bin/activate
export PYTHONUSERBASE=$HOME/local/$SYS_TYPE
Load appropriate compiler modules. The same compiler that was used to compile libROM should be used here. The setup below uses gcc as an example.
module load python/3.9.12
module load gcc/12.1.1-magic
module load mkl/2022.1.0
Activate virtualenv from above and set the correct python site path:
source $HOME/local/$SYS_TYPE/bin/activate
export PYTHONUSERBASE=$HOME/local/$SYS_TYPE
Install SWIG to your local Python environment. The system provided version of swig is too old and will cause a compile error if used.
pip install swig==4.1.1
Download and compile PyMFEM:
git clone https://github.com/mfem/PyMFEM.git
cd PyMFEM
python setup.py install -v --with-parallel --with-gslib --CC=gcc --CXX=g++ --MPICC=mpicc --MPICXX=mpic++ --with-lapack
Download and compile pylibROM:
git clone --recurse-submodules https://github.com/LLNL/pylibROM.git
cd pylibROM/
If libROM has already been compiled, specify the path (recommended):
pip install -v ./ --global-option="--librom_dir=/path/to/pre-installed-libROM"
If libROM has not been compiled, pylibROM can compile it:
pip install -v ./
Note: the following environment variable currently needs to be set to avoid runtime errors when running PyMFEM and pylibROM.
export LD_PRELOAD=/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-12.1.1/lib/libmpi.so:$MKLROOT/lib/intel64/libmkl_core.so:$MKLROOT/lib/intel64/libmkl_sequential.so:$MKLROOT/lib/intel64/libmkl_intel_lp64.so:$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so:$MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so:$MKLROOT/lib/intel64/libmkl_gf_lp64.so:$MKLROOT/lib/intel64/libmkl_gnu_thread.so
First, load appropriate modules. The same compiler that was used to compile libROM should be used here. The module list below uses gcc as an example.
module load python/3.9.12
module load gcc/12.1.1-magic
module load mkl/2022.1.0
Install SWIG to your local Python environment and add it to your path. The system provided version of swig is too old and will cause a compile error if used.
pip install --user swig==4.1.1
export PATH=~/.local/bin:$PATH
Download and compile PyMFEM:
git clone https://github.com/mfem/PyMFEM.git
cd PyMFEM
python3 setup.py install --with-parallel --with-gslib --user --CC=gcc --CXX=g++ --MPICC=mpicc --MPICXX=mpic++ --with-lapack
Download and compile pylibROM:
git clone --recurse-submodules https://github.com/llnl/pylibROM.git
cd pylibROM/
If libROM has already been compiled, specify the path (recommended):
pip install ./ --global-option="--librom_dir=/path/to/pre-installed-libROM"
If libROM has not been compiled, pylibROM can compile it:
pip install ./
Note: Since Lassen is a different architecture, care must be taken to ensure that the Python environment, user packages, and all dependencies (libROM, MFEM, pyMFEM, etc) are compiled for the correct architecture. The section below demonstrates creating a virtualenv where local user packages are installed into a different prefix based on the LC provided environment variable $SYS_TYPE. This will allow local installations of PyMFEM/pylibROM to be used across different LC machines.
module load python/3.8.2
Install a newer virtualenv since the system version gives a site error. Need at least virtualenv >= 20.0.0.
pip install --user virtualenv==20.0.0
export PATH=$HOME/.local/bin:$PATH
Create a unique virtualenv for the current system type
virtualenv --system-site-packages $HOME/local/$SYS_TYPE
export PATH=$HOME/local/$SYS_TYPE/bin:$PATH
Activate system specific virtualenv
source $HOME/local/$SYS_TYPE/bin/activate
Use which python to verify that the correct python is being used.
Note: After the virtualenv is created once, it does not need to be repeated. Only activating the virtualenv (source $HOME/local/$SYS_TYPE/bin/activate) needs to be done in the future.
Activate virtualenv from above and load appropriate modules. The same compiler that was used to compile libROM should be used here. The module list below uses gcc as an example.
module load python/3.8.2
module load gcc/10.2.1
Install SWIG to your local Python environment and add it to your path. The system provided version of swig is too old and will cause a compile error if used.
pip install swig==4.1.1
Download and compile PyMFEM:
git clone https://github.com/mfem/PyMFEM.git
cd PyMFEM
python setup.py install --with-parallel --with-gslib --CC=gcc --CXX=g++ --MPICC=mpicc --MPICXX=mpic++
Download and compile pylibROM:
git clone --recurse-submodules https://github.com/llnl/pylibROM.git
cd pylibROM/
If libROM has already been compiled, specify the path (recommended):
pip install ./ --global-option="--librom_dir=/path/to/pre-installed-libROM"
If libROM has not been compiled, pylibROM can compile it:
pip install ./