[Trilinos-Users] [EXTERNAL] How to force trillinos to link with libiomp5 instead of libgomp?

Elliott, James John jjellio at sandia.gov
Tue Jun 27 16:22:41 EDT 2017 

Hi,

This isn't really a Trilinos issue, but more about how to use GCC to link intel's OpenMP. I would start by adding the Intel OpenMP library to the Blas/Lapack required libraries. This will ensure the correct library is found and linked consistently. I generally avoid using CMAKE_EXE_LINKER_FLAGS.

Try setting your Lapack and BLAS libs as:

  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread" \

**You may or may not need to specify pthread**

Next, try setting:
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread;-Wl,--as-needed,-lgomp,--no-as-needed" \

You can also experiment with this directly. E.g., cd to the directory with the example/unit test, and edit CMakeFiles/<blah.exe>/link.txt.  Add -Wl,--as-needed,-lgomp,--no-as-needed.  Then run make to have it trigger a relink.

No promise that will work ( untested advice! ). I have had to do something similar on a few systems. If you have issues adding the linker arguments to the Library Name's CMake variable, then you can directly specify them using CMAKE_EXE_LINKER_FLAGS.

James

________________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Wen Yan <wenyan4work at gmail.com>
Sent: Tuesday, June 27, 2017 1:49 PM
To: trilinos-users at trilinos.org
Subject: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5       instead of libgomp?

Dear Trilinos Users,

I was wondering how to force Trillinos to link with the intel openmp library, instead of the gnu openmp library libgomp, when using gnu compilers.

My current toolchain on a Mac is gcc7 from home-brew with openmp support and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt, using the gnu runtime libgomp, but with this openmp runtime the MKL can only work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a Mac we only have libmkl_intel_thread.dylib.

So I was wondering how to force Trilinos to link with intel openmp runtime, libiomp5, for all the static and dynamic libraries, and all the test examples. Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?

My current attempts are not successful.
I was using this configure with:
  -D TPL_ENABLE_MKL:BOOL=ON \
  -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
  -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
  -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
  -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
  -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \

But found that e.g. the test exe
./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
        @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
        @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        … other libs

Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to it. It is usually a big problem where those two libraries are simultaneously linked.

Then when I type "make test", all test cases failed, and I got the message in the test log file:

dyld: Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
  Reason: image not found

But in “echo $DYLD_LIBRARY_PATH”, I can find all the correct paths, and libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.

Could you please let me know how to forbid linking with libgomp?


Thanks a lot,
Wen Yan

PS1 :
This is the cmake command options:
/Applications/CMake.app/Contents/bin/ccmake  \
  -D CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \
  -D CMAKE_BUILD_TYPE:STRING=RELEASE \
  -D TPL_ENABLE_MPI:BOOL=ON \
  -D MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \
  -D MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \
  -D Trilinos_ENABLE_CXX11:BOOL=ON \
  -D Trilinos_ENABLE_Belos:BOOL=ON \
  -D Trilinos_ENABLE_Fortran:BOOL=OFF \
  -D Trilinos_ENABLE_TESTS:BOOL=ON \
  -D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
  -D Trilinos_ENABLE_OpenMP:BOOL=ON \
  -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
  -D TPL_ENABLE_Pthread:BOOL=ON \
  -D TPL_ENABLE_MKL:BOOL=ON \
  -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
  -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
  -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
  -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
  -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
  -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
  -D Ifpack2_ENABLE_EXAMPLES:BOOL=ON \
  -D Kokkos_ENABLE_HWLOC:BOOL=OFF \
  -D TPL_ENABLE_HWLOC:BOOL=OFF \
$EXTRA_ARGS \
$TRILINOS_PATH


PS2: The libraries linked to a test exe:
otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
        @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
        /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
        /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
        /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

PS3: The test log file:
1/105 Testing: Belos_ValidParameters_MPI_1
1/105 Test: Belos_ValidParameters_MPI_1
Command: "/Users/wyan/local/bin/mpiexec" "-np" "1" "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe" "--add-xsl-header"
Directory: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList
"Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
Output:
----------------------------------------------------------
dyld: Library not loaded: @rpath/libiomp5.dylib
  Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
  Reason: image not found
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node ccbmac035 exited on signal 6 (Abort trap: 6).
--------------------------------------------------------------------------
<end of output>
Test time =   2.07 sec

PS4:
echo $DYLD_LIBRARY_PATH
/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib

_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
https://trilinos.org/mailman/listinfo/trilinos-users

More information about the Trilinos-Users mailing list