[Trilinos-Users] [EXTERNAL] How to force trillinos to link with libiomp5 instead of libgomp?
Wen Yan
wenyan4work at gmail.com
Fri Jun 30 09:12:57 EDT 2017
Hi James and Roscoe,
Thanks for your help with this non-Trilinos problem. I got the rpath issue from the link in Roscoe’s email.
About the openmp issue, after some web search I found that probably the only clean solution for the iomp5 issue on a Mac is to use Clang/LLVM, and they offer an option “-fopenmp=libiomp5” to manually tell clang to link the executable to intel’s openmp lib.
I can compile the program like this:
wyan$ /usr/local/opt/llvm/bin/clang++ -fopenmp=libiomp5 ./test.cpp -L/opt/intel/lib -L/usr/local/opt/llvm/lib/
wyan$ otool -L ./a.out
./a.out:
@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
However with clang, the -L/opt/intel/lib option must be put in front of llvm’s internal lib path, because llvm (at least the llvm I got from homebrew) includes a libiomp5 which is a symbolic link to its internal libgomp:
wyan$ ls -al /usr/local/opt/llvm/lib/libiomp5.dylib
lrwxr-xr-x 1 wyan admin 12 Jun 12 21:46 /usr/local/opt/llvm/lib/libiomp5.dylib -> libomp.dylib
Their libomp actually has a unfixed bug reported here, from the old intel openmp system which is the upstream of clang/llvm’s openmp:
https://software.intel.com/en-us/forums/intel-c-compiler/topic/291343 <https://software.intel.com/en-us/forums/intel-c-compiler/topic/291343>
Use clang with the real new libiomp5 from intel eliminates this bug.
Thanks a lot, and I hope this may help those people who have no choice but to use a Mac as their development environment.
Wen Yan
> On Jun 28, 2017, at 6:34 PM, Elliott, James John <jjellio at sandia.gov> wrote:
>
> How annoying.
>
> Unfortunately, I have no experience with MacOS. Something looks unusual about your configuration though.
>
> It looks like cmake is generating a link line that wants to statically link your TPL, which is probably why it is not adding RPATH information for it.
>
> One thing to test is setting:
> -D BUILD_SHARED_LIBS:BOOL=ON
>
> Your CMake output seems to be missing the part I was looking for. Typically, when I configure with TPLs, CMake will add output like the following. What you want to check is that CMake is grabbing the correct *full* paths to your libs. Based on link.txt, it is clear that it does have the correct path.
>
> Processing enabled TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
> -- BLAS_LIBRARY_NAMES='openblas;pthread'
> -- Searching for libs in BLAS_LIBRARY_DIRS='/home/jjellio/install/OpenBLAS/lib'
> -- Searching for a lib in the set "openblas":
> -- Searching for lib 'openblas' ...
> -- Found lib '/home/jjellio/install/OpenBLAS/lib/libopenblas.so'
> -- Searching for a lib in the set "pthread":
> -- Searching for lib 'pthread' ...
> -- Found lib '/usr/lib64/libpthread.so'
> -- TPL_BLAS_LIBRARIES='/home/jjellio/install/OpenBLAS/lib/libopenblas.so;/usr/lib64/libpthread.so'
>
> From: Wen Yan <wenyan4work at gmail.com>
> Sent: Wednesday, June 28, 2017 4:08 PM
> To: Elliott, James John
> Cc: trilinos-users at trilinos.org; Bartlett, Roscoe A
> Subject: Re: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
>
> Hi James,
>
> Thanks for the explanation, and I did a lot of trial-and-errors.
>
>> CMake should dump some output as it processes each TPL showing the path of the libraries found. Is $MKLROOT set? Is mkl_rt located in MKLROOT/lib ?
>
>
> I am sure that $MKLROOT and $MKLROOT/lib are correctly set:
> build_test wyan$ ls $MKLROOT/lib/libmkl_rt*
> /opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib/libmkl_rt.dylib
>
>> Can you please post the output from CMake? E.g., cmake ... options ... |& tee cmake_output.log.
>> Based on the output from your first email, I am guessing CMake is failing to obtain the correct paths for mkl_rt.
>
>
> I did a test build with the following setting:
> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
> -D LAPACK_LIBRARY_NAMES:STRING="mkl_rt;iomp5;pthread;-Wl,--as-needed,-lgomp,--no-as-needed;-Wl,--verbose" \
> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
> -D CMAKE_LINKER:STRING="/usr/local/g++-7” \
>
> My mpicxx/mpicc are manually compiled with g++-7 with openmpi-2.1.1, and the paths are set correctly. The output of the cmake is really long and is attached to the end of this email.
>
>> I recommend seeing if adding '-Wl,--as-needed,library,--no-as-needed' will fix this for you. This tells the linker to only link 'library' if it fails to find the symbols in all other libraries. Since iomp5 should provide a full OpenMP implementation then linking only iomp5 should be needed. The linker should choose to omit the library. You can see exactly what happens at link time, by added -Wl,--verbose.
>
>
> I added the link lines as you recommended in your first email at the LAPACK line, but the '-Wl,--as-needed,library,--no-as-needed’ part is not recognized by cmake. I found the link.txt for test exe:
>
> build_test wyan$ cat ./packages/tpetra/core/test/Block/CMakeFiles/TpetraCore_ExpBlockOps.dir/link.txt
> /Users/wyan/local/bin/mpicxx -std=c++11 -fopenmp -O3 -DNDEBUG -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/TpetraCore_ExpBlockOps.dir/ExpBlockOps.cpp.o CMakeFiles/TpetraCore_ExpBlockOps.dir/__/__/__/__/teuchos/core/test/UnitTest/Teuchos_StandardUnitTestMain.cpp.o -o TpetraCore_ExpBlockOps.exe ../../ext/libtpetraext.a ../../inout/libtpetrainout.a ../../src/libtpetra.a ../../../tsqr/src/libkokkostsqr.a ../../../kernels/src/libtpetrakernels.a ../../../classic/LinAlg/libtpetraclassiclinalg.a ../../../classic/NodeAPI/libtpetraclassicnodeapi.a ../../../classic/src/libtpetraclassic.a ../../../../epetra/src/libepetra.a ../../../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.a ../../../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.a ../../../../teuchos/remainder/src/libteuchosremainder.a ../../../../teuchos/numerics/src/libteuchosnumerics.a /opt/intel/lib/libiomp5.dylib /usr/lib/libpthread.dylib /opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib/libmkl_rt.dylib ../../../../teuchos/comm/src/libteuchoscomm.a ../../../../teuchos/parameterlist/src/libteuchosparameterlist.a ../../../../teuchos/core/src/libteuchoscore.a ../../../../kokkos/algorithms/src/libkokkosalgorithms.a ../../../../kokkos/containers/src/libkokkoscontainers.a ../../../../kokkos/core/src/libkokkoscore.a /usr/lib/libdl.dylib
>
> The '-Wl,--as-needed,library,--no-as-needed’ part is not added to the link line, and the exe is linked simultaneously to both iomp5 (through rpath) and gomp (full path):
> build_test wyan$ otool -L ./packages/tpetra/core/test/Block/TpetraCore_ExpBlockOps.exe
> ./packages/tpetra/core/test/Block/TpetraCore_ExpBlockOps.exe:
> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
> /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
> /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
> /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
> /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
> And the rpath for iomp5 is not embedded into this exe file. The LC_RPATH in the exe contains only the gcc lib folder '/usr/local/opt/gcc/lib'
>
>
> Then if I do "make test”, all tests fail because they cannot find libiomp5 and mkl_rt
> dyld: Library not loaded: @rpath/libmkl_rt.dylib
> Referenced from: /Users/wyan/software/Trilinos/build_test/packages/tpetra/kernels/unit_test/TpetraKernels_CrsMatrixUnitTest.exe
> Reason: image not found
>
> But if directly call the executable from the bash shell, it runs successfully:
> build_test wyan$ ./packages/tpetra/core/test/Block/TpetraCore_ExpBlockOps.exe
> Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ccbmac035.flatironinstitute.org <http://ccbmac035.flatironinstitute.org/> and rank 0!
> Sorting tests by group name then by the order they were added ... (time = 5.01e-06)
>
> Running unit tests ...
>
> 0. LittleBlockOps_double_GEMV_UnitTest ... [Passed] (0.00397 sec)
> 1. LittleBlockOps_int_GEMV_UnitTest ... [Passed] (0.00238 sec)
> 2. LittleBlockOps_longlong_GEMV_UnitTest ... [Passed] (0.0022 sec)
>
> Total Time: 0.00862 sec
>
> Summary: total = 3, run = 3, passed = 3, failed = 0
>
> End Result: TEST PASSED
>
> ——————————————————————————
>
> Now if I manually edit the link.txt to add the '-Wl,--as-needed,-lgomp,--no-as-needed’ at the end of the link line, and trigger the make process, it fails:
> '[ 51%] Linking CXX executable TpetraCore_ExpBlockOps.exe
> ld: unknown option: --as-needed'
>
> I googled something about this and found that this is a common problem on Mac. The Mac default /usr/bin/ld does not support this option, and I could not find a working gnu ld on a Mac. Linking with g++-7 on a Mac seems to internally call the Mac default ld. The gnu binutils from homebrew contains almost everything except gnu-ld.
>
> Now it seems to be Mac’s fault. Maybe we need some workaround for a Mac based system without manually messing with rpath?
>
> Thanks,
> Wen Yan
>
>
> -- Found PythonInterp: /usr/bin/python (found suitable version "2.7.10", minimum required is "2.6")
> -- Setting Trilinos_ENABLE_EXAMPLES=ON because Trilinos_ENABLE_TESTS=ON
> -- Setting CMAKE_C_COMPILER=${MPI_C_COMPILER}
> -- Setting CMAKE_CXX_COMPILER=${MPI_CXX_COMPILER}
> -- The C compiler identification is GNU 7.1.0
> -- Checking whether C compiler has -isysroot
> -- Checking whether C compiler has -isysroot - yes
> -- Checking whether C compiler supports OSX deployment target flag
> -- Checking whether C compiler supports OSX deployment target flag - yes
> -- Check for working C compiler: /Users/wyan/local/bin/mpicc
> -- Check for working C compiler: /Users/wyan/local/bin/mpicc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- The CXX compiler identification is GNU 7.1.0
> -- Checking whether CXX compiler has -isysroot
> -- Checking whether CXX compiler has -isysroot - yes
> -- Checking whether CXX compiler supports OSX deployment target flag
> -- Checking whether CXX compiler supports OSX deployment target flag - yes
> -- Check for working CXX compiler: /Users/wyan/local/bin/mpicxx
> -- Check for working CXX compiler: /Users/wyan/local/bin/mpicxx -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- Looking for C++ include sys/time.h
> -- Looking for C++ include sys/time.h - found
> -- Looking for C++ include time.h
> -- Looking for C++ include time.h - found
> -- Looking for C++ include stdint.h
> -- Looking for C++ include stdint.h - found
> -- Looking for C++ include inttypes.h
> -- Looking for C++ include inttypes.h - found
> -- Found Perl: /usr/bin/perl (found version "5.18.2")
> -- Performing Test CXX11_FLAGS_COMPILE_RESULT_0
> -- Performing Test CXX11_FLAGS_COMPILE_RESULT_0 - Success
> -- Performing Test CXX11_CONSECUTIVE_RIGHT_ANGLE_BRACKETS
> -- Performing Test CXX11_CONSECUTIVE_RIGHT_ANGLE_BRACKETS - Success
> -- Performing Test CXX11_AUTOTYPEDVARIABLES
> -- Performing Test CXX11_AUTOTYPEDVARIABLES - Success
> -- Performing Test CXX11_LAMBDAS
> -- Performing Test CXX11_LAMBDAS - Success
> -- Try OpenMP C flag = [-fopenmp]
> -- Performing Test OpenMP_FLAG_DETECTED
> -- Performing Test OpenMP_FLAG_DETECTED - Success
> -- Try OpenMP CXX flag = [-fopenmp]
> -- Performing Test OpenMP_FLAG_DETECTED
> -- Performing Test OpenMP_FLAG_DETECTED - Success
> -- Found OpenMP: -fopenmp
> -- Performing Test MATH_LIBRARY_IS_SUPPLIED
> -- Performing Test MATH_LIBRARY_IS_SUPPLIED - Success
> -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN
> -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN - Failed
> -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN
> -- Performing Test FINITE_VALUE_HAVE_STD_ISNAN - Success
> -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF
> -- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF - Failed
> -- Performing Test FINITE_VALUE_HAVE_STD_ISINF
> -- Performing Test FINITE_VALUE_HAVE_STD_ISINF - Success
> -- Found Doxygen: /usr/local/bin/doxygen (found version "1.8.13")
> -- Looking for pthread.h
> -- Looking for pthread.h - found
> -- Looking for pthread_create
> -- Looking for pthread_create - found
> -- Found Threads: TRUE
> -- Performing Test HAVE_GCC_ABI_DEMANGLE
> -- Performing Test HAVE_GCC_ABI_DEMANGLE - Success
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT - Failed
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT_APPLE_VECLIB_BUGFIX
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT_APPLE_VECLIB_BUGFIX - Failed
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT_DOUBLE_RETURN
> -- Performing Test HAVE_TEUCHOS_BLASFLOAT_DOUBLE_RETURN - Failed
> -- Performing Test HAVE_TEUCHOS_LAPACKLARND
> -- Performing Test HAVE_TEUCHOS_LAPACKLARND - Failed
> -- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR
> -- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR - Success
> -- C++ compiler supports __attribute__((constructor)) syntax
> -- Performing Test HAVE_CXX_ATTRIBUTE_WEAK
> -- Performing Test HAVE_CXX_ATTRIBUTE_WEAK - Failed
> -- C++ compiler does NOT support __attribute__((weak)) syntax and testing weak functions
> -- Performing Test HAVE_CXX_PRAGMA_WEAK
> -- Performing Test HAVE_CXX_PRAGMA_WEAK - Failed
> -- C++ compiler does NOT support #pragma weak syntax and testing weak functions
> -- Performing Test CXX_COMPLEX_BLAS_WORKS
> -- Performing Test CXX_COMPLEX_BLAS_WORKS - Failed
> -- Performing Test HAVE_FIXABLE_COMPLEX_BLAS_PROBLEM
> -- Performing Test HAVE_FIXABLE_COMPLEX_BLAS_PROBLEM - Failed
> -- NOTE: Kokkos::Serial is ON (the CMake option Kokkos_ENABLE_Serial is ON), but the corresponding Tpetra Node type is disabled. If you want to enable instantiation and use of Kokkos::Serial in Tpetra, please also set the CMake option Tpetra_INST_SERIAL:BOOL=ON. If you use the Kokkos::Serial Node type in Tpetra without doing this, you will get link errors!
> -- Tpetra execution space availability (ON means available):
> -- - Serial: OFF
> -- - Threads: OFF
> -- - OpenMP: ON
> -- - Cuda: OFF
> -- Setting default Node to Kokkos::Compat::KokkosOpenMPWrapperNode.
> -- Not enabling Tpetra/RTI
> -- TpetraCore: Processing ETI / test support
> -- Enabled Scalar types: int|long long|double
> -- Enabled LocalOrdinal types: int
> -- Enabled GlobalOrdinal types: int|long long
> -- Enabled Node types: Kokkos::Compat::KokkosOpenMPWrapperNode
> -- Set of enabled types, before exclusions: S={int} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={int} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long};S={long long} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={long long} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long}
> -- TpetraKernels: Processing ETI / test support
> -- Enabled Scalar types: int|long long|double
> -- Enabled LocalOrdinal types: int
> -- Enabled Device types: Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>
> -- Set of enabled types, before exclusions: S={int} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>};S={long long} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>};S={double} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>}
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /Users/wyan/software/Trilinos/build_test2
>
>
>
>
>
>
>
>
>
>
>
>> On Jun 28, 2017, at 3:59 PM, Elliott, James John <jjellio at sandia.gov <mailto:jjellio at sandia.gov>> wrote:
>>
>> Hi Wen,
>>
>> Can you please post the output from CMake? E.g., cmake ... options ... |& tee cmake_output.log.
>> Based on the output from your first email, I am guessing CMake is failing to obtain the correct paths for mkl_rt. CMake should dump some output as it processes each TPL showing the path of the libraries found. Is $MKLROOT set? Is mkl_rt located in MKLROOT/lib ?
>>
>>
>> The first part of your message about DYLD_LIBRARY_PATH is why you want to specify libraries to CMake the way I recommended. Trilinos uses CMake, and so the questions really center around getting CMake to do the right thing.
>>
>> When you specify -Wl,-rpath,/path/, you are instructing the linker to add '/path/' to the runtime path of the executable. If done correctly, then you do not need DYLD_LIBRARY_PATH, because the executable has the correct library paths already.
>>
>> When you set CMAKE_EXE_LINKER_FLAGS, you are sidestepping CMake's built in rpath handling code. This matters because you do not always want to build dynamically linked executables (I rarely do), and systems can handle libraries differently.
>>
>> It could also be that CMake is not setting rpath correctly for MacOS. (Rpath issues do come up). CMake has a full page of details about this: https://cmake.org/Wiki/CMake_RPATH_handling <https://cmake.org/Wiki/CMake_RPATH_handling>. I recommend avoiding setting anything related to RPATH though - CMake should do this for you.
>>
>>
>>
>> The second comment is much more complicated. You generally need the same optimization flags used at compile time to be used at link time. This matters more for Intel, IBM, and Cray compilers, but can also matter for GNU compilers. I recommend seeing if adding '-Wl,--as-needed,library,--no-as-needed' will fix this for you. This tells the linker to only link 'library' if it fails to find the symbols in all other libraries. Since iomp5 should provide a full OpenMP implementation then linking only iomp5 should be needed. The linker should choose to omit the library. You can see exactly what happens at link time, by added -Wl,--verbose.
>>
>> The easiest way to troubleshoot this is to manually edit the link.txt file, add the above flags at the *end* of the link line, and see if the generated executable has the correct RPATH and if GOMP is being linked. If GOMP is linked, then add the verbose flag and find out why. I think GOMP is being linked because the path to iomp5 is not being correctly discovered or set.
>>
>>
>>
>> James
>>
>>
>> ________________________________________
>> From: Wen Yan <wenyan4work at gmail.com <mailto:wenyan4work at gmail.com>>
>> Sent: Wednesday, June 28, 2017 7:55 AM
>> To: Elliott, James John
>> Cc: trilinos-users at trilinos.org <mailto:trilinos-users at trilinos.org>
>> Subject: Re: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
>>
>> Hi James,
>>
>> Thanks a lot for the information. I found that my problem actually involves two separate issues.
>>
>> ——————————
>> The first one is the DYLD_LIBRARY_PATH settings on a Mac. Due to some recent Mac updates the DYLD_LIBRARY_PATH cannot be found in the env.
>>
>> This command gives nothing: “env | grep DYLD”.
>> This command shows the path: “echo $DYLD_LIBRARY_PATH”.
>>
>> So when I invoke “make test”, it seems that the ctest program sometimes failed to set correct DYLD path and failed all tests.
>>
>> If I set
>> -D CMAKE_EXE_LINKER_FLAGS:STRING=" -L$MKLROOT/lib -Wl,-rpath,$MKLROOT/lib -lmkl_rt -L/opt/intel/lib -Wl,-rpath,/opt/intel/lib -liomp5” \
>> to embed the rpath to the exe files, then "make test” runs correctly.
>>
>> ——————————
>> The second part is the linking. I found that CMakeFiles/<blah.exe>/link.txt starts with this:
>> “/Users/wyan/local/bin/mpicxx -std=c++11 -fopenmp -O3 -DNDEBUG “
>> When -fopenmp is passed to the linker, gcc will link to libgomp so both libiomp5 and libgomp are linked simultaneous to those two openmp runtime libraries.
>>
>> I can manually modify the link.txt to remove the “-fopenmp” flag and thus the libgomp dependency, and the test passed correctly. However I didn’t find a universal setting to remove that link flag completely for all test files. Maybe it is something internal to the build system of Trilinos.
>>
>> ——————————
>>
>> For now my problem is somehow solved, but it would be great if we can have an explicit option to control the linking flags of Trilions.
>>
>> Thanks a lot,
>> Wen Yan
>>
>>
>>
>>
>>
>>> On Jun 27, 2017, at 4:22 PM, Elliott, James John <jjellio at sandia.gov <mailto:jjellio at sandia.gov>> wrote:
>>>
>>> Hi,
>>>
>>> This isn't really a Trilinos issue, but more about how to use GCC to link intel's OpenMP. I would start by adding the Intel OpenMP library to the Blas/Lapack required libraries. This will ensure the correct library is found and linked consistently. I generally avoid using CMAKE_EXE_LINKER_FLAGS.
>>>
>>> Try setting your Lapack and BLAS libs as:
>>>
>>> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
>>> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread" \
>>>
>>> **You may or may not need to specify pthread**
>>>
>>> Next, try setting:
>>> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/;/opt/intel/lib" \
>>> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt;iomp5;pthread;-Wl,--as-needed,-lgomp,--no-as-needed" \
>>>
>>> You can also experiment with this directly. E.g., cd to the directory with the example/unit test, and edit CMakeFiles/<blah.exe>/link.txt. Add -Wl,--as-needed,-lgomp,--no-as-needed. Then run make to have it trigger a relink.
>>>
>>> No promise that will work ( untested advice! ). I have had to do something similar on a few systems. If you have issues adding the linker arguments to the Library Name's CMake variable, then you can directly specify them using CMAKE_EXE_LINKER_FLAGS.
>>>
>>> James
>>>
>>> ________________________________________
>>> From: Trilinos-Users <trilinos-users-bounces at trilinos.org <mailto:trilinos-users-bounces at trilinos.org>> on behalf of Wen Yan <wenyan4work at gmail.com <mailto:wenyan4work at gmail.com>>
>>> Sent: Tuesday, June 27, 2017 1:49 PM
>>> To: trilinos-users at trilinos.org <mailto:trilinos-users at trilinos.org>
>>> Subject: [EXTERNAL] [Trilinos-Users] How to force trillinos to link with libiomp5 instead of libgomp?
>>>
>>> Dear Trilinos Users,
>>>
>>> I was wondering how to force Trillinos to link with the intel openmp library, instead of the gnu openmp library libgomp, when using gnu compilers.
>>>
>>> My current toolchain on a Mac is gcc7 from home-brew with openmp support and Intel MKL 2017. I can compile Trilinos with MKL single dynamic link -lmkl_rt, using the gnu runtime libgomp, but with this openmp runtime the MKL can only work (all tests passed) with MKL_NUM_THREADS=1 (sequential mode). This is because on a mac the MKL does not offer libmkl_gnu_thread as on a Linux. On a Mac we only have libmkl_intel_thread.dylib.
>>>
>>> So I was wondering how to force Trilinos to link with intel openmp runtime, libiomp5, for all the static and dynamic libraries, and all the test examples. Which LINKER_FLAGS_RELEASE should I pass in to the cmake system?
>>>
>>> My current attempts are not successful.
>>> I was using this configure with:
>>> -D TPL_ENABLE_MKL:BOOL=ON \
>>> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
>>> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
>>> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
>>> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
>>> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
>>> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
>>> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
>>>
>>> But found that e.g. the test exe
>>> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
>>> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
>>> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
>>> … other libs
>>>
>>> Which shows libiomp5 is dynamically linked to it, but libgomp is also linked to it. It is usually a big problem where those two libraries are simultaneously linked.
>>>
>>> Then when I type "make test", all test cases failed, and I got the message in the test log file:
>>>
>>> dyld: Library not loaded: @rpath/libiomp5.dylib
>>> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
>>> Reason: image not found
>>>
>>> But in “echo $DYLD_LIBRARY_PATH”, I can find all the correct paths, and libiomp5.dylib is located in the first path in DYLD_LIBRARY_PATH.
>>>
>>> Could you please let me know how to forbid linking with libgomp?
>>>
>>>
>>> Thanks a lot,
>>> Wen Yan
>>>
>>> PS1 :
>>> This is the cmake command options:
>>> /Applications/CMake.app/Contents/bin/ccmake \
>>> -D CMAKE_INSTALL_PREFIX:FILEPATH="/Users/wyan/local/" \
>>> -D CMAKE_BUILD_TYPE:STRING=RELEASE \
>>> -D TPL_ENABLE_MPI:BOOL=ON \
>>> -D MPI_BASE_DIR:FILEPATH="/Users/wyan/local" \
>>> -D MPI_EXEC:FILEPATH="/Users/wyan/local/bin" \
>>> -D Trilinos_ENABLE_CXX11:BOOL=ON \
>>> -D Trilinos_ENABLE_Belos:BOOL=ON \
>>> -D Trilinos_ENABLE_Fortran:BOOL=OFF \
>>> -D Trilinos_ENABLE_TESTS:BOOL=ON \
>>> -D Trilinos_ENABLE_EXAMPLES:BOOL=ON \
>>> -D Trilinos_ENABLE_OpenMP:BOOL=ON \
>>> -D Trilinos_ENABLE_Ifpack2:BOOL=ON \
>>> -D TPL_ENABLE_Pthread:BOOL=ON \
>>> -D TPL_ENABLE_MKL:BOOL=ON \
>>> -D MKL_INCLUDE_DIRS:FILEPATH="$MKLROOT/include" \
>>> -D MKL_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D MKL_LIBRARY_NAMES:STRING="mkl_rt" \
>>> -D BLAS_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D BLAS_LIBRARY_NAMES:STRING="mkl_rt" \
>>> -D LAPACK_LIBRARY_DIRS:FILEPATH="$MKLROOT/lib/" \
>>> -D LAPACK_LIBRARY_NAMES:FILEPATH="mkl_rt" \
>>> -D CMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="-L/opt/intel/lib/ -Wl,-rpath,/opt/intel/lib -liomp5" \
>>> -D CMAKE_CXX_FLAGS_RELEASE:STRING="-O3 -march=native" \
>>> -D CMAKE_C_FLAGS_RELEASE:STRING="-O3 -march=native" \
>>> -D Ifpack2_ENABLE_EXAMPLES:BOOL=ON \
>>> -D Kokkos_ENABLE_HWLOC:BOOL=OFF \
>>> -D TPL_ENABLE_HWLOC:BOOL=OFF \
>>> $EXTRA_ARGS \
>>> $TRILINOS_PATH
>>>
>>>
>>> PS2: The libraries linked to a test exe:
>>> otool -L ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe
>>> ./packages/belos/test/GCRODR/Belos_gcrodr_complex_hb.exe:
>>> @rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
>>> @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
>>> /Users/wyan/local/lib/libmpi_cxx.20.dylib (compatibility version 31.0.0, current version 31.0.0)
>>> /Users/wyan/local/lib/libmpi.20.dylib (compatibility version 31.0.0, current version 31.1.0)
>>> /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.23.0)
>>> /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
>>> /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
>>>
>>> PS3: The test log file:
>>> 1/105 Testing: Belos_ValidParameters_MPI_1
>>> 1/105 Test: Belos_ValidParameters_MPI_1
>>> Command: "/Users/wyan/local/bin/mpiexec" "-np" "1" "/Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe" "--add-xsl-header"
>>> Directory: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList
>>> "Belos_ValidParameters_MPI_1" start time: Jun 27 15:08 EDT
>>> Output:
>>> ----------------------------------------------------------
>>> dyld: Library not loaded: @rpath/libiomp5.dylib
>>> Referenced from: /Users/wyan/software/Trilinos/build_iomp5/packages/belos/doc/parameterList/Belos_ValidParameters.exe
>>> Reason: image not found
>>> --------------------------------------------------------------------------
>>> mpiexec noticed that process rank 0 with PID 0 on node ccbmac035 exited on signal 6 (Abort trap: 6).
>>> --------------------------------------------------------------------------
>>> <end of output>
>>> Test time = 2.07 sec
>>>
>>> PS4:
>>> echo $DYLD_LIBRARY_PATH
>>> /opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib/intel64:/opt/intel/compilers_and_libraries_2017.4.181/mac/tbb/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/compiler/lib:/opt/intel/compilers_and_libraries_2017.4.181/mac/mkl/lib
>>>
>>> _______________________________________________
>>> Trilinos-Users mailing list
>>> Trilinos-Users at trilinos.org <mailto:Trilinos-Users at trilinos.org>
>>> https://trilinos.org/mailman/listinfo/trilinos-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20170630/0562f5b1/attachment-0001.html>
More information about the Trilinos-Users
mailing list