[Trilinos-Users] [EXTERNAL] How to force trillinos to link with libiomp5 instead of libgomp?
Wen Yan
wenyan4work at gmail.com
Wed Jun 28 09:55:38 EDT 2017
Hi James,
Thanks a lot for the information. I found that my problem actually involves two separate issues.
——————————
The first one is the DYLD_LIBRARY_PATH settings on a Mac. Due to some recent Mac updates the DYLD_LIBRARY_PATH cannot be found in the env.
This command gives nothing: “env | grep DYLD”.
This command shows the path: “echo $DYLD_LIBRARY_PATH”.
So when I invoke “make test”, it seems that the ctest program sometimes failed to set correct DYLD path and failed all tests.
If I set
-D CMAKE_EXE_LINKER_FLAGS:STRING=" -L$MKLROOT/lib -Wl,-rpath,$MKLROOT/lib -lmkl_rt -L/opt/intel/lib -Wl,-rpath,/opt/intel/lib -liomp5” \
to embed the rpath to the exe files, then "make test” runs correctly.
——————————
The second part is the linking. I found that CMakeFiles/<blah.exe>/link.txt starts with this:
“/Users/wyan/local/bin/mpicxx -std=c++11 -fopenmp -O3 -DNDEBUG “
When -fopenmp is passed to the linker, gcc will link to libgomp so both libiomp5 and libgomp are linked simultaneous to those two openmp runtime libraries.
I can manually modify the link.txt to remove the “-fopenmp” flag and thus the libgomp dependency, and the test passed correctly. However I didn’t find a universal setting to remove that link flag completely for all test files. Maybe it is something internal to the build system of Trilinos.
——————————
For now my problem is somehow solved, but it would be great if we can have an explicit option to control the linking flags of Trilions.
Thanks a lot,
Wen Yan
> On Jun 27, 2017, at 4:22 PM, Elliott, James John <jjellio at sandia.gov> wrote:
>
> Hi,
>
> This isn't really a Trilinos issue, but more about how to use GCC to link intel's OpenMP. I would start by adding the Intel OpenMP library to the Blas/Lapack required libraries. This will ensure the correct library is found and linked consistently. I generally avoid using CMAKE_EXE_LINKER_FLAGS.
>
> Try setting your Lapack and BLAS libs as:
>
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread" \
>
> **You may or may not need to specify pthread**
>
> Next, try setting:
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread;-Wl,--as-needed,-lgomp,--no-as-needed" \
>
> You can also experiment with this directly. E.g., cd to the directory with the example/unit test, and edit CMakeFiles/<blah.exe>/link.txt. Add -Wl,--as-needed,-lgomp,--no-as-needed. Then run make to have it trigger a relink.
>
> No promise that will work ( untested advice! ). I have had to do something similar on a few systems. If you have issues adding the linker arguments to the Library Name's CMake variable, then you can directly specify them using CMAKE_EXE_LINKER_FLAGS.
>
> James
>
> ________________________________________
> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Wen Yan <wenyan4work at gmail.com>
> Sent: Tuesday, June 27, 2017 1:49 PM
> To: trilinos-users at trilinos.org
> Subject: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
>
> Dear Trilinos Users,
>
> I was wondering how to force Trillinos to link with the intel openmp library, instead of the gnu openmp library libgomp, when using gnu compilers.
>
> My current toolchain on a Mac is gcc7 from home-brew with openmp support and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt, using the gnu runtime libgomp, but with this openmp runtime the MKL can only work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a Mac we only have libmkl_intel_thread.dylib.
>
> So I was wondering how to force Trilinos to link with intel openmp runtime, libiomp5, for all the static and dynamic libraries, and all the test examples. Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?
>
> My current attempts are not successful.
> I was using this configure with:
> -D TPL_ENABLE_MKL:BOOL=ON \
> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
>
> But found that e.g. the test exe
> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
> … other libs
>
> Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to it. It is usually a big problem where those two libraries are simultaneously linked.
>
> Then when I type "make test", all test cases failed, and I got the message in the test log file:
>
> dyld: Library not loaded: @rpath/libiomp5.dylib
> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
> Reason: image not found
>
> But in “echo $DYLD_LIBRARY_PATH”, I can find all the correct paths, and libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.
>
> Could you please let me know how to forbid linking with libgomp?
>
>
> Thanks a lot,
> Wen Yan
>
> PS1 :
> This is the cmake command options:
> /Applications/CMake.app/Contents/bin/ccmake \
> -D CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \
> -D CMAKE_BUILD_TYPE:STRING=RELEASE \
> -D TPL_ENABLE_MPI:BOOL=ON \
> -D MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \
> -D MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \
> -D Trilinos_ENABLE_CXX11:BOOL=ON \
> -D Trilinos_ENABLE_Belos:BOOL=ON \
> -D Trilinos_ENABLE_Fortran:BOOL=OFF \
> -D Trilinos_ENABLE_TESTS:BOOL=ON \
> -D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
> -D Trilinos_ENABLE_OpenMP:BOOL=ON \
> -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
> -D TPL_ENABLE_Pthread:BOOL=ON \
> -D TPL_ENABLE_MKL:BOOL=ON \
> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D Ifpack2_ENABLE_EXAMPLES:BOOL=ON \
> -D Kokkos_ENABLE_HWLOC:BOOL=OFF \
> -D TPL_ENABLE_HWLOC:BOOL=OFF \
> $EXTRA_ARGS \
> $TRILINOS_PATH
>
>
> PS2: The libraries linked to a test exe:
> otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
> /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
> /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
> /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
>
> PS3: The test log file:
> 1/105 Testing: Belos_ValidParameters_MPI_1
> 1/105 Test: Belos_ValidParameters_MPI_1
> Command: "/Users/wyan/local/bin/mpiexec" "-np" "1" "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe" "--add-xsl-header"
> Directory: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList
> "Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
> Output:
> ----------------------------------------------------------
> dyld: Library not loaded: @rpath/libiomp5.dylib
> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
> Reason: image not found
> --------------------------------------------------------------------------
> mpiexec noticed that process rank 0 with PID 0 on node ccbmac035 exited on signal 6 (Abort trap: 6).
> --------------------------------------------------------------------------
> <end of output>
> Test time = 2.07 sec
>
> PS4:
> echo $DYLD_LIBRARY_PATH
> /opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list