[Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?

Tue Jun 27 15:49:42 EDT 2017

Dear Trilinos Users,

I was wondering how to force Trillinos to link with the intel openmp library, instead of the gnu openmp library libgomp, when using gnu compilers.

My current toolchain on a Mac is gcc7 from home-brew with openmp support and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt, using the gnu runtime libgomp, but with this openmp runtime the MKL can only work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a Mac we only have libmkl_intel_thread.dylib.

So I was wondering how to force Trilinos to link with intel openmp runtime, libiomp5, for all the static and dynamic libraries, and all the test examples. Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?

My current attempts are not successful. 
I was using this configure with:
  -D TPL_ENABLE_MKL:BOOL=ON \
  -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
  -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
  -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
  -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
  -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \

But found that e.g. the test exe
./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	… other libs

Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to it. It is usually a big problem where those two libraries are simultaneously linked.

Then when I type "make test", all test cases failed, and I got the message in the test log file:

dyld: Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
  Reason: image not found

But in “echo $DYLD_LIBRARY_PATH”, I can find all the correct paths, and libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.

Could you please let me know how to forbid linking with libgomp?

Thanks a lot,
Wen Yan

PS1 :
This is the cmake command options:
/Applications/CMake.app/Contents/bin/ccmake  \
  -D CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \
  -D CMAKE_BUILD_TYPE:STRING=RELEASE \
  -D TPL_ENABLE_MPI:BOOL=ON \
  -D MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \
  -D MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \
  -D Trilinos_ENABLE_CXX11:BOOL=ON \
  -D Trilinos_ENABLE_Belos:BOOL=ON \
  -D Trilinos_ENABLE_Fortran:BOOL=OFF \
  -D Trilinos_ENABLE_TESTS:BOOL=ON \
  -D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
  -D Trilinos_ENABLE_OpenMP:BOOL=ON \
  -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
  -D TPL_ENABLE_Pthread:BOOL=ON \
  -D TPL_ENABLE_MKL:BOOL=ON \
  -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
  -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
  -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
  -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
  -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D Ifpack2_ENABLE_EXAMPLES:BOOL=ON \
  -D Kokkos_ENABLE_HWLOC:BOOL=OFF \
  -D TPL_ENABLE_HWLOC:BOOL=OFF \
$EXTRA_ARGS \
$TRILINOS_PATH 

PS2: The libraries linked to a test exe:
otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
	/Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
	/usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
	/usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	/usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

PS3: The test log file:
1/105 Testing: Belos_ValidParameters_MPI_1
1/105 Test: Belos_ValidParameters_MPI_1
Command: "/Users/wyan/local/bin/mpiexec" "-np" "1" "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe" "--add-xsl-header"
Directory: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList
"Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
Output:
----------------------------------------------------------
dyld: Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
  Reason: image not found
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node ccbmac035 exited on signal 6 (Abort trap: 6).
--------------------------------------------------------------------------
<end of output>
Test time =   2.07 sec

PS4: 
echo $DYLD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib