Recipe: Compile OpenFOAM* with Intel® Compiler for Intel® Xeon® Scalable Processors

A stepwise guide to compile and run OpenFOAM* with Intel® Compiler for Intel®  Xeon® Scalable Processor

Objective

Demonstrate the steps involved in compiling OpenFOAM* software with Intel® Compiler for running it on Intel Xeon Scalable Processor. This recipe describes how to get OpenFOAM* and corresponding third party software and build all the code with Intel® Compiler and how to run an example on Intel® Xeon® Scalable Processors.

Introduction

OpenFOAM* (Open source Field Operation And Manipulation) is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of computational fluid dynamics (CFD). It is a unique, and highly extensible, software development kit (“devkit”) for CFD, consisting of 1 million lines of C++ code. OpenFOAM includes hundreds of robust CFD applications built from the devkit, which can be extended and customized, quickly and conveniently. The applications are used to preform CFD simulations by industry, academic, and research and government institutes, across all fields of engineering and science

OpenFOAM is distributed as open source code by OpenFOAM Foundation, under the terms of the GNU Public License. OpenFOAM is developed by multiple contributors led by Henry Weller, creator of OpenFOAM and Director of CFD Direct and The OpenFOAM Foundation. The current version can be downloaded at https://openfoam.org/download/. Links are also included for Ubuntu Linux, Other Linux, Windows* and macOS version of the software.

Compiling OpenFOAM Intel® Compiler

Specifications

Code Access

Step 1. Downloading OpenFOAM*

mkdir OpenFOAM

wget -P OpenFOAM "http://download.openfoam.org/source/4-1" -O OpenFOAM-4.1.tgz
tar -xzf OpenFOAM/OpenFOAM-4.1.tgz

cd OpenFOAM
mv OpenFOAM-4.x-version-4.1 OpenFOAM-4.1

Step 2. Downloading Third Party Software :

In order to build OpenFOAM, we need to download Third Party Software which has a basic requirement of having cmake-3.2.1 and above . If you do not have cmake-3.2.1, then please download and install it before compiling any of the Third Party Software. Download the following Third Party Software:

•  CGAL-4.8
•  boost_1_55_0
•  gmp-5.1.2
•  mpfr-3.1.2
•  metis-5.1.0

wget "http://download.openfoam.org/third-party/4-1" -O ThirdParty-4.1.tgz 
tar -xzf OpenFOAM/ThirdParty-4.1.tgz

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1
mv ThirdParty-4.x-version-4.1 ThirdParty-4.1

mkdir download

wget -P download "https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-4.8/CGAL-4.8.tar.xz"

wget -P download "http://sourceforge.net/projects/boost/files/boost/1.55.0/boost_1_55_0.tar.bz2"

wget –P download "https://ftp.gnu.org/gnu/gmp/gmp-5.1.2.tar.bz2"  

wget –P download "http://www.mpfr.org/mpfr-3.1.2/mpfr-3.1.2.tar.gz"  

wget -P download "http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/metis-5.1.0.tar.gz"

tar -xJf download/CGAL-4.8.tar.xz 
tar -xjf download/boost_1_55_0.tar.bz2 
tar –zxf download/metis-5.1.0.tar.gz 
tar -xjvf download/gmp-5.1.2.tar.bz2 
tar -xzvf download/mpfr-3.1.2.tar.gz 

Build Directions

The most fundamental way to efficiently utilize the resources in modern processors is to write code that can run in vector mode by taking advantage of special hardware like vector registers and SIMD (Single Instruction Multiple Data) instructions. Data parallelism in the algorithm/code is exploited in this stage of the optimization process. The new Intel® Xeon® scalable processor features 512-bit wide vector registers. The new Intel® Advanced Vector Extensions 512 (Intel® AVX-512) instruction set architecture (ISA), offers support for vector-level parallelism, which allows the software to use two vector processing units (each capable of simultaneously processing 16 single precision (32-bit) or 8 double precision (64-bit) floating point numbers) per core. Taking advantage of these hardware and software features is the key to optimal use of the Intel® Xeon® scalable processor, which can be leveraged by compiler option -xCORE-AVX512.

Step 1. Building OpenFOAM* with Intel® Compiler is similar to a normal OpenFOAM build. Need to make a few changes in the config settings and config mpi along with the bashrc. An example build will include the following steps while compiling it for 64 bit architecture:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/OpenFOAM-4.1 
1. vi etc/config.sh/mpi 
	case "$WM_MPLIB" in - INTELMPI 
	export MPI_ROOT=/opt/intel/compilers_and_libraries_2018.0.128/linux/mpi/intel64 

2. vi etc/config.sh/settings 
	a. case $WM_ARCH in Linux 
		case `uname -m` in > x86_64 
			case $WM_ARCH_OPTION in 
				i ) 32 
				ii) 64 
					> export WM_CC='icc' 
					> export WM_CXX='icpc' 
					> export WM_CFLAGS='-xCORE-AVX512 -O2 -no-prec-div -m64 -fPIC'
					> export WM_CXXFLAGS='-xCORE-AVX512 -O2 -no-prec-div -m64 -fPIC' 

3. vi etc/bashrc 
	> export FOAM_INST_DIR=/<YOUR_OPENFOAM_LOCATION>/OpenFOAM 
	> export WM_COMPILER=Icc 
	> export WM_MPLIB=INTELMPI 

Step 2. Create a setup file, which you need to source before any compilation:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
vi setup
	> source /opt/intel/compilers_and_libraries_2018.0.128/linux/bin/compilervars.sh intel64

	> source /opt/intel/impi/2018.0.128/bin64/mpivars.sh
	> source ./OpenFOAM-4.1/etc/bashrc 
	> cd ThirdParty-4.1

	> sed -i -e 's/\(boost_version=\)boost-system/\1boost_1_55_0/' OpenFOAM-4.1/etc/config.sh/CGAL

	> sed -i -e 's/\(cgal_version=\)cgal-system/\1CGAL-4.8/' OpenFOAM-4.1/etc/config.sh/CGAL
	
	> cd ..
	> source ./OpenFOAM-4.1/etc/bashrc WM_LABEL_SIZE=64 WM_COMPILER_TYPE=system FOAMY_HEX_MESH=yes

Step 3. Building Third Party Softwares:

Building gmp with Intel® Compiler:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1/gmp-5.1.2

./configure --prefix=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2 CC=icc CXX=icpc CFLAGS=-xCORE-AVX512 CXXFLAGS=-xCORE-AVX512
 
mkdir platforms/linux64/gmp-5.1.2
make
make check
make install

Building mpfr with Intel® Compiler:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1/mpfr-3.1.2

./configure --prefix=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/mpfr-3.1.2 CC=icc CXX=icpc CFLAGS=-xCORE-AVX512 CXXFLAGS=-xCORE-AVX512 --with-gmp-include=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/include --with-gmp-lib=<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/lib

mkdir platforms/linux64/mpfr-3.1.2
make
make check
make install

Building CGAL with Intel® Compiler:

In order to build CGAL, we need to build the boost library as well. In the makefile of CGAL, while calling the bootstap.sh, replace "bjam" with "b2" and use intel toolset to build the library (line no.161). Add the Intel C/C++ Compiler and optimization flags as -xCORE-AVX512 (line no 277). Following are the changes:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd ThirdParty-4.1
vi makeCGAL
	> --with-toolset=intel-linux \
	> && ./b2 -j $WM_NCOMPPROCS install \
	> ....
	> ....
	> -DCMAKE_C_COMPILER=icc \
	> -DCMAKE_CXX_COMPILER=icpc \
	> -DCMAKE_C_FLAGS=-xCORE-AVX512 \
	> -DCMAKE_CXX_FLAGS=-xCORE-AVX512 \

In order to make CGAL with gmp-5.1.2 and mpfr-3.1.2 library, we need to have gmp-5.1.2 and mpfr-3.1.2 libraries, which were created at the location:

cd Thirdparty-4.1.x/platforms/gmp-5.1.2/lib
cd Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib

While compiling CGAL, the library files are searched under lib64 folder, hence we need to rename the lib folder to lib64 as follows and then make the CGAL library:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM

mv Thirdparty-4.1.x/platforms/gmp-5.1.2/lib Thirdparty-4.1.x/platforms/gmp-5.1.2/lib64

mv Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib Thirdparty-4.1.x/platforms/mpfr-5.1.2/lib64

vi OpenFOAM-4.1.x/wmake/rules/General/CGAL
	> CGAL_INC = -I<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/mpfr-3.1.2/include\ 
-I<YOUR_OPENFOAM_LOCATION>/OpenFOAM/ThirdParty-4.1/platforms/linux64/gmp-5.1.2/include

	> CGAL_LIBS = -L<YOUR_OPENFOAM_LOCATION>OpenFOAM/platforms/linux64/mpfr-3.1.2/lib$(WM_COMPILER_LIB_ARCH) \
-L<YOUR_OPENFOAM_LOCATION>/OpenFOAM/platforms/linux64/gmp-5.1.2/lib$(WM_COMPILER_LIB_ARCH) \

source setup
cd Thirdparty-4.1.x
./makeCGAL CGAL-4.8 boost_1_55_0 gmp-5.1.2 mpfr-3.1.2

Building Scotch library with Intel Compiler

•  Remove scotch building steps from OpenFOAM-4.1.x/Allwmake
•  Make changes to SCOTCH_Num from int to int64_t in /Thirdparty-4.1.x/scotch_6.0.3/include/scotch.h
•  Remove previous simlink to /Thirdparty-4.1.x/scotch_6.0.3/Makefile.inc
•  Make changes to /Thirdparty-4.1.x/scotch_6.0.3/Make.inc/Makefile.inc.x86-64_pc_linux2.icc
  • CCS = icc
  • CCP = mpiicc
  • CCD = mpiicc
  • #add –xCORE-AVX512 –fPIC flag in CFLAGS

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
cd Thirdparty-4.1.x/scotch_6.0.3/src/
rm Makefile.inc
vi Make.inc/Makefile.inc.x86-64_pc_linux2.icc
	> CCS = icc
	> CCP = mpiicc
	> CCD = mpiicc
	> CFALG = -xCORE-AVX512 -fPIC (additional flags)

ln –s Make.inc/Makefile.inc.x86-64_pc_linux2.icc Makefile.inc

make scotch
make prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3 libdir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/lib install

make ptscotch
make prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3 libdir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/lib/intel64 includedir=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/scotch_6.0.3/include/intel64 install

Building metis-5.1.0 library with Intel Compiler

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
cd Thirdparty-4.1.x/matis-5.1.0
vi Makefile
	> openmp = -qopenmp
	> cc = mpiicc

make config prefix=$WM_THIRD_PARTY_DIR/platforms/linux64IccDPInt64/metis-5.1.0
make
make install

Step 4. Building OpenFOAM-4.1.x with Intel Compiler:

We have already updated the cOpt and c++Opt with -xCORE-AVX512 in the step no. 2. Now we need to provide correct IntelMPI path in the mplibINTELMPI file. Replace include64 and lib64 with include and lib

• PINC       = -isystem $(MPI_ARCH_PATH)/include64
• PLIBS      = -L$(MPI_ARCH_PATH)/lib64 –lmpi

 cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM/OpenFOAM-4.1.x
vi wmake/rules/linux64Icc/mplibINTELMPI
	> PINC  = -isystem $(MPI_ARCH_PATH)/include 
	> PLIBS = -L$(MPI_ARCH_PATH)/lib -lmpi
cd ..
source setup
export WM_NCOMPPROCS=8
./Allwmake -j 8 2>&1 | tee output.log

You can check output.log if the compilation is done successful. Once compilation is done, you can run the example and check if it is working fine.

Test with tutorials

You can check your OpenFOAM installation is working fine or not by running all tutorials. This will take a lot of time to run all of them, one by one.

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd OpenFOAM-4.1/tutorials/
./Allrun 	

However, you can also just run one of the case in the tutorial, e.g.:

cd <YOUR_OPENFOAM_LOCATION>/OpenFOAM
source setup
cd OpenFOAM-4.1/tutorials/incompressible/simpleFoam/motorbike/
./Allrun