[Trilinos-Users] [EXTERNAL] How to force trillinos to link with libiomp5 instead of libgomp?

Wed Jun 28 17:09:52 EDT 2017

For guidance and behavior of RPATH, see:

   https://trilinos.org/docs/files/TrilinosBuildReference.html#setting-install-rpath

Dr. Roscoe A. Bartlett, PhD
http://www.cs.sandia.gov/cr-rabartl
Sandia National Laboratories

> -----Original Message-----
> From: Trilinos-Users [mailto:trilinos-users-bounces at trilinos.org] On Behalf Of
> Elliott, James John
> Sent: Wednesday, June 28, 2017 4:00 PM
> To: Wen Yan <wenyan4work at gmail.com>
> Cc: trilinos-users at trilinos.org
> Subject: Re: [Trilinos-Users] [EXTERNAL] How to force trillinos to link with
> libiomp5 instead of libgomp?
> 
> Hi Wen,
> 
> Can you please post the output from CMake? E.g., cmake ... options ... |& tee
> cmake_output.log.
> Based on the output from your first email, I am guessing CMake is failing to
> obtain the correct paths for mkl_rt.  CMake should dump some output as it
> processes each TPL showing the path of the libraries found. Is $MKLROOT set? Is
> mkl_rt located in MKLROOT/lib ?
> 
> 
> The first part of your message about DYLD_LIBRARY_PATH is why you want to
> specify libraries to CMake the way I recommended. Trilinos uses CMake, and so
> the questions really center around getting CMake to do the right thing.
> 
> When you specify -Wl,-rpath,/path/, you are instructing the linker to add '/path/'
> to the runtime path of the executable. If done correctly, then you do not need
> DYLD_LIBRARY_PATH, because the executable has the correct library paths
> already.
> 
> When you set CMAKE_EXE_LINKER_FLAGS, you are sidestepping CMake's built in
> rpath handling code.  This matters because you do not always want to build
> dynamically linked executables (I rarely do), and systems can handle libraries
> differently.
> 
> It could also be that CMake is not setting rpath correctly for MacOS. (Rpath
> issues do come up). CMake has a full page of details about this:
> https://cmake.org/Wiki/CMake_RPATH_handling.  I recommend avoiding
> setting anything related to RPATH though - CMake should do this for you.
> 
> 
> 
> The second comment is much more complicated.  You generally need the same
> optimization flags used at compile time to be used at link time.  This matters
> more for Intel, IBM, and Cray compilers, but can also matter for GNU compilers.
> I recommend seeing if adding '-Wl,--as-needed,library,--no-as-needed' will fix
> this for you.  This tells the linker to only link 'library' if it fails to find the symbols
> in all other libraries.  Since iomp5 should provide a full OpenMP implementation
> then linking only iomp5 should be needed. The linker should choose to omit the
> library.  You can see exactly what happens at link time, by added -Wl,--verbose.
> 
> The easiest way to troubleshoot this is to manually edit the link.txt file, add the
> above flags at the *end* of the link line, and see if the generated executable has
> the correct RPATH and if GOMP is being linked.  If GOMP is linked, then add the
> verbose flag and find out why. I think GOMP is being linked because the path to
> iomp5 is not being correctly discovered or set.
> 
> 
> 
> James
> 
> 
> ________________________________________
> From: Wen Yan <wenyan4work at gmail.com>
> Sent: Wednesday, June 28, 2017 7:55 AM
> To: Elliott, James John
> Cc: trilinos-users at trilinos.org
> Subject: Re: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with
> libiomp5   instead of libgomp?
> 
> Hi James,
> 
> Thanks a lot for the information. I found that my problem actually involves two
> separate issues.
> 
> ----------
> The first one is the DYLD_LIBRARY_PATH settings on a Mac. Due to some recent
> Mac updates the DYLD_LIBRARY_PATH cannot be found in the env.
> 
> This command gives nothing: "env | grep DYLD".
> This command shows the path: "echo $DYLD_LIBRARY_PATH".
> 
> So when I invoke "make test", it seems that the ctest program sometimes failed
> to set correct DYLD path and failed all tests.
> 
> If I set
>  -D CMAKE_EXE_LINKER_FLAGS:STRING=" -L$MKLROOT/lib -Wl,-
> rpath,$MKLROOT/lib -lmkl_rt -L/opt/intel/lib -Wl,-rpath,/opt/intel/lib -liomp5" \
> to embed the rpath to the exe files, then "make test" runs correctly.
> 
> ----------
> The second part is the linking. I found that CMakeFiles/<blah.exe>/link.txt starts
> with this:
> "/Users/wyan/local/bin/mpicxx   -std=c++11 -fopenmp -O3 -DNDEBUG "
> When -fopenmp is passed to the linker, gcc will link to libgomp so both libiomp5
> and libgomp are linked simultaneous to those two openmp runtime libraries.
> 
> I can manually modify the link.txt to remove the "-fopenmp" flag and thus the
> libgomp dependency, and the test passed correctly. However I didn't find a
> universal setting to remove that link flag completely for all test files. Maybe it is
> something internal to the build system of Trilinos.
> 
> ----------
> 
> For now my problem is somehow solved, but it would be great if we can have an
> explicit option to control the linking flags of Trilions.
> 
> Thanks a lot,
> Wen Yan
> 
> 
> 
> 
> 
> > On Jun 27, 2017, at 4:22 PM, Elliott, James John <jjellio at sandia.gov> wrote:
> >
> > Hi,
> >
> > This isn't really a Trilinos issue, but more about how to use GCC to link intel's
> OpenMP. I would start by adding the Intel OpenMP library to the Blas/Lapack
> required libraries. This will ensure the correct library is found and linked
> consistently. I generally avoid using CMAKE_EXE_LINKER_FLAGS.
> >
> > Try setting your Lapack and BLAS libs as:
> >
> >  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \  -D
> > LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread" \
> >
> > **You may or may not need to specify pthread**
> >
> > Next, try setting:
> >  -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \  -D
> > LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread;-Wl,--as-needed,-
> l
> > gomp,--no-as-needed" \
> >
> > You can also experiment with this directly. E.g., cd to the directory with the
> example/unit test, and edit CMakeFiles/<blah.exe>/link.txt.  Add -Wl,--as-
> needed,-lgomp,--no-as-needed.  Then run make to have it trigger a relink.
> >
> > No promise that will work ( untested advice! ). I have had to do something
> similar on a few systems. If you have issues adding the linker arguments to the
> Library Name's CMake variable, then you can directly specify them using
> CMAKE_EXE_LINKER_FLAGS.
> >
> > James
> >
> > ________________________________________
> > From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf
> > of Wen Yan <wenyan4work at gmail.com>
> > Sent: Tuesday, June 27, 2017 1:49 PM
> > To: trilinos-users at trilinos.org
> > Subject: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with
> libiomp5       instead of libgomp?
> >
> > Dear Trilinos Users,
> >
> > I was wondering how to force Trillinos to link with the intel openmp library,
> instead of the gnu openmp library libgomp, when using gnu compilers.
> >
> > My current toolchain on a Mac is gcc7 from home-brew with openmp support
> and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt,
> using the gnu runtime libgomp, but with this openmp runtime the MKL can only
> work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is
> because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a
> Mac we only have libmkl_intel_thread.dylib.
> >
> > So I was wondering how to force Trilinos to link with intel openmp runtime,
> libiomp5, for all the static and dynamic libraries, and all the test examples.
> Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?
> >
> > My current attempts are not successful.
> > I was using this configure with:
> >  -D TPL_ENABLE_MKL:BOOL=ON \
> >  -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \  -D
> > MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > MKL_LIBRARY_NAMES:STRING="mkl_rt" \  -D
> > BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > BLAS_LIBRARY_NAMES:STRING="mkl_rt" \  -D
> > LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \  -D
> > CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/
> > -Wl,-rpath,/opt/intel/lib -liomp5" \  -D
> > CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \  -D
> > CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
> >
> > But found that e.g. the test exe
> > ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> >        @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> >        @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> >        /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0,
> current version 2.0.0)
> >        ... other libs
> >
> > Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to
> it. It is usually a big problem where those two libraries are simultaneously linked.
> >
> > Then when I type "make test", all test cases failed, and I got the message in
> the test log file:
> >
> > dyld: Library not loaded: @rpath/libiomp5.dylib  Referenced from:
> > /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameter
> > List/Belos_ValidParameters.exe
> >  Reason: image not found
> >
> > But in "echo $DYLD_LIBRARY_PATH", I can find all the correct paths, and
> libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.
> >
> > Could you please let me know how to forbid linking with libgomp?
> >
> >
> > Thanks a lot,
> > Wen Yan
> >
> > PS1 :
> > This is the cmake command options:
> > /Applications/CMake.app/Contents/bin/ccmake  \  -D
> > CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \  -D
> > CMAKE_BUILD_TYPE:STRING=RELEASE \  -D TPL_ENABLE_MPI:BOOL=ON \  -D
> > MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \  -D
> > MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \  -D
> > Trilinos_ENABLE_CXX11:BOOL=ON \  -D Trilinos_ENABLE_Belos:BOOL=ON \
> > -D Trilinos_ENABLE_Fortran:BOOL=OFF \  -D
> > Trilinos_ENABLE_TESTS:BOOL=ON \  -D
> Trilinos_ENABLE_EXAMPLES:BOOL=ON \
> > -D Trilinos_ENABLE_OpenMP:BOOL=ON \  -D
> > Trilinos_ENABLE_Ifpack2:BOOL=ON \  -D TPL_ENABLE_Pthread:BOOL=ON \  -
> D
> > TPL_ENABLE_MKL:BOOL=ON \  -D
> > MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \  -D
> > MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > MKL_LIBRARY_NAMES:STRING="mkl_rt" \  -D
> > BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > BLAS_LIBRARY_NAMES:STRING="mkl_rt" \  -D
> > LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \  -D
> > LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \  -D
> > CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/
> > -Wl,-rpath,/opt/intel/lib -liomp5" \  -D
> > CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \  -D
> > CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \  -D
> > Ifpack2_ENABLE_EXAMPLES:BOOL=ON \  -D
> Kokkos_ENABLE_HWLOC:BOOL=OFF \
> > -D TPL_ENABLE_HWLOC:BOOL=OFF \ $EXTRA_ARGS \ $TRILINOS_PATH
> >
> >
> > PS2: The libraries linked to a test exe:
> > otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
> > ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
> >        @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> >        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1238.60.2)
> >        @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> >        /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0,
> current version 31.0.0)
> >        /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current
> version 31.1.0)
> >        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0,
> current version 7.23.0)
> >        /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0,
> current version 2.0.0)
> >        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version
> > 1.0.0, current version 1.0.0)
> >
> > PS3: The test log file:
> > 1/105 Testing: Belos_ValidParameters_MPI_1
> > 1/105 Test: Belos_ValidParameters_MPI_1
> > Command: "/Users/wyan/local/bin/mpiexec" "-np" "1"
> "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterLi
> st/Belos_ValidParameters.exe" "--add-xsl-header"
> > Directory:
> > /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameter
> > List "Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
> > Output:
> > ----------------------------------------------------------
> > dyld: Library not loaded: @rpath/libiomp5.dylib  Referenced from:
> > /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameter
> > List/Belos_ValidParameters.exe
> >  Reason: image not found
> > ----------------------------------------------------------------------
> > ---- mpiexec noticed that process rank 0 with PID 0 on node ccbmac035
> > exited on signal 6 (Abort trap: 6).
> > ----------------------------------------------------------------------
> > ----
> > <end of output>
> > Test time =   2.07 sec
> >
> > PS4:
> > echo $DYLD_LIBRARY_PATH
> > /opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/in
> > tel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/i
> > ntel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compile
> > rs_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_
> > libraries_2017.4.181/mac/mkl/lib
> >
> > _______________________________________________
> > Trilinos-Users mailing list
> > Trilinos-Users at trilinos.org
> > https://trilinos.org/mailman/listinfo/trilinos-users
> 
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users