[Trilinos-Users] [EXTERNAL] How to force trillinos to link with libiomp5 instead of libgomp?
Elliott, James John
jjellio at sandia.gov
Wed Jun 28 15:59:57 EDT 2017
Hi Wen,
Can you please post the output from CMake? E.g., cmake ... options ... |& tee cmake_output.log.
Based on the output from your first email, I am guessing CMake is failing to obtain the correct paths for mkl_rt. CMake should dump some output as it processes each TPL showing the path of the libraries found. Is $MKLROOT set? Is mkl_rt located in MKLROOT/lib ?
The first part of your message about DYLD_LIBRARY_PATH is why you want to specify libraries to CMake the way I recommended. Trilinos uses CMake, and so the questions really center around getting CMake to do the right thing.
When you specify -Wl,-rpath,/path/, you are instructing the linker to add '/path/' to the runtime path of the executable. If done correctly, then you do not need DYLD_LIBRARY_PATH, because the executable has the correct library paths already.
When you set CMAKE_EXE_LINKER_FLAGS, you are sidestepping CMake's built in rpath handling code. This matters because you do not always want to build dynamically linked executables (I rarely do), and systems can handle libraries differently.
It could also be that CMake is not setting rpath correctly for MacOS. (Rpath issues do come up). CMake has a full page of details about this: https://cmake.org/Wiki/CMake_RPATH_handling. I recommend avoiding setting anything related to RPATH though - CMake should do this for you.
The second comment is much more complicated. You generally need the same optimization flags used at compile time to be used at link time. This matters more for Intel, IBM, and Cray compilers, but can also matter for GNU compilers. I recommend seeing if adding '-Wl,--as-needed,library,--no-as-needed' will fix this for you. This tells the linker to only link 'library' if it fails to find the symbols in all other libraries. Since iomp5 should provide a full OpenMP implementation then linking only iomp5 should be needed. The linker should choose to omit the library. You can see exactly what happens at link time, by added -Wl,--verbose.
The easiest way to troubleshoot this is to manually edit the link.txt file, add the above flags at the *end* of the link line, and see if the generated executable has the correct RPATH and if GOMP is being linked. If GOMP is linked, then add the verbose flag and find out why. I think GOMP is being linked because the path to iomp5 is not being correctly discovered or set.
James
________________________________________
From: Wen Yan <wenyan4work at gmail.com>
Sent: Wednesday, June 28, 2017 7:55 AM
To: Elliott, James John
Cc: trilinos-users at trilinos.org
Subject: Re: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
Hi James,
Thanks a lot for the information. I found that my problem actually involves two separate issues.
——————————
The first one is the DYLD_LIBRARY_PATH settings on a Mac. Due to some recent Mac updates the DYLD_LIBRARY_PATH cannot be found in the env.
This command gives nothing: “env | grep DYLD”.
This command shows the path: “echo $DYLD_LIBRARY_PATH”.
So when I invoke “make test”, it seems that the ctest program sometimes failed to set correct DYLD path and failed all tests.
If I set
-D CMAKE_EXE_LINKER_FLAGS:STRING=" -L$MKLROOT/lib -Wl,-rpath,$MKLROOT/lib -lmkl_rt -L/opt/intel/lib -Wl,-rpath,/opt/intel/lib -liomp5” \
to embed the rpath to the exe files, then "make test” runs correctly.
——————————
The second part is the linking. I found that CMakeFiles/<blah.exe>/link.txt starts with this:
“/Users/wyan/local/bin/mpicxx -std=c++11 -fopenmp -O3 -DNDEBUG “
When -fopenmp is passed to the linker, gcc will link to libgomp so both libiomp5 and libgomp are linked simultaneous to those two openmp runtime libraries.
I can manually modify the link.txt to remove the “-fopenmp” flag and thus the libgomp dependency, and the test passed correctly. However I didn’t find a universal setting to remove that link flag completely for all test files. Maybe it is something internal to the build system of Trilinos.
——————————
For now my problem is somehow solved, but it would be great if we can have an explicit option to control the linking flags of Trilions.
Thanks a lot,
Wen Yan
> On Jun 27, 2017, at 4:22 PM, Elliott, James John <jjellio at sandia.gov> wrote:
>
> Hi,
>
> This isn't really a Trilinos issue, but more about how to use GCC to link intel's OpenMP. I would start by adding the Intel OpenMP library to the Blas/Lapack required libraries. This will ensure the correct library is found and linked consistently. I generally avoid using CMAKE_EXE_LINKER_FLAGS.
>
> Try setting your Lapack and BLAS libs as:
>
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread" \
>
> **You may or may not need to specify pthread**
>
> Next, try setting:
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread;-Wl,--as-needed,-lgomp,--no-as-needed" \
>
> You can also experiment with this directly. E.g., cd to the directory with the example/unit test, and edit CMakeFiles/<blah.exe>/link.txt. Add -Wl,--as-needed,-lgomp,--no-as-needed. Then run make to have it trigger a relink.
>
> No promise that will work ( untested advice! ). I have had to do something similar on a few systems. If you have issues adding the linker arguments to the Library Name's CMake variable, then you can directly specify them using CMAKE_EXE_LINKER_FLAGS.
>
> James
>
> ________________________________________
> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Wen Yan <wenyan4work at gmail.com>
> Sent: Tuesday, June 27, 2017 1:49 PM
> To: trilinos-users at trilinos.org
> Subject: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
>
> Dear Trilinos Users,
>
> I was wondering how to force Trillinos to link with the intel openmp library, instead of the gnu openmp library libgomp, when using gnu compilers.
>
> My current toolchain on a Mac is gcc7 from home-brew with openmp support and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt, using the gnu runtime libgomp, but with this openmp runtime the MKL can only work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a Mac we only have libmkl_intel_thread.dylib.
>
> So I was wondering how to force Trilinos to link with intel openmp runtime, libiomp5, for all the static and dynamic libraries, and all the test examples. Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?
>
> My current attempts are not successful.
> I was using this configure with:
> -D TPL_ENABLE_MKL:BOOL=ON \
> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
>
> But found that e.g. the test exe
> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
> … other libs
>
> Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to it. It is usually a big problem where those two libraries are simultaneously linked.
>
> Then when I type "make test", all test cases failed, and I got the message in the test log file:
>
> dyld: Library not loaded: @rpath/libiomp5.dylib
> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
> Reason: image not found
>
> But in “echo $DYLD_LIBRARY_PATH”, I can find all the correct paths, and libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.
>
> Could you please let me know how to forbid linking with libgomp?
>
>
> Thanks a lot,
> Wen Yan
>
> PS1 :
> This is the cmake command options:
> /Applications/CMake.app/Contents/bin/ccmake \
> -D CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \
> -D CMAKE_BUILD_TYPE:STRING=RELEASE \
> -D TPL_ENABLE_MPI:BOOL=ON \
> -D MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \
> -D MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \
> -D Trilinos_ENABLE_CXX11:BOOL=ON \
> -D Trilinos_ENABLE_Belos:BOOL=ON \
> -D Trilinos_ENABLE_Fortran:BOOL=OFF \
> -D Trilinos_ENABLE_TESTS:BOOL=ON \
> -D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
> -D Trilinos_ENABLE_OpenMP:BOOL=ON \
> -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
> -D TPL_ENABLE_Pthread:BOOL=ON \
> -D TPL_ENABLE_MKL:BOOL=ON \
> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D Ifpack2_ENABLE_EXAMPLES:BOOL=ON \
> -D Kokkos_ENABLE_HWLOC:BOOL=OFF \
> -D TPL_ENABLE_HWLOC:BOOL=OFF \
> $EXTRA_ARGS \
> $TRILINOS_PATH
>
>
> PS2: The libraries linked to a test exe:
> otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
> /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
> /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
> /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
>
> PS3: The test log file:
> 1/105 Testing: Belos_ValidParameters_MPI_1
> 1/105 Test: Belos_ValidParameters_MPI_1
> Command: "/Users/wyan/local/bin/mpiexec" "-np" "1" "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe" "--add-xsl-header"
> Directory: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList
> "Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
> Output:
> ----------------------------------------------------------
> dyld: Library not loaded: @rpath/libiomp5.dylib
> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
> Reason: image not found
> --------------------------------------------------------------------------
> mpiexec noticed that process rank 0 with PID 0 on node ccbmac035 exited on signal 6 (Abort trap: 6).
> --------------------------------------------------------------------------
> <end of output>
> Test time = 2.07 sec
>
> PS4:
> echo $DYLD_LIBRARY_PATH
> /opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
More information about the Trilinos-Users
mailing list