[Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
Hysom, David A.
hysom1 at llnl.gov
Wed Feb 22 07:59:54 EST 2017
>Is there a specific need for 128bit bit precision?
Not by me. But there's apparently some place (in trilinos?) where
there's a macro that uses it, which is causing the problem.
We have some loadable clang modules that claim to support cuda,
but I couldn't get them to work; I'll check with my local support IT guys
on that. I also tried compiling with intel (v. 17), but it failed some of
Trilinos' tests for c++11 compatibility.
________________________________
From: Elliott, James John <jjellio at sandia.gov>
Sent: Tuesday, February 21, 2017 11:58:11 PM
To: Hysom, David A.; trilinos-users at trilinos.org
Cc: Trott, Christian Robert (-EXP); Hoemmen, Mark
Subject: Re: [Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
David,
Ignore my prior statements. I read their pull request incorrectly.
https://svn.boost.org/trac/boost/ticket/11852
Says they tested with CUDA 7.5 and float128 is still not working with nvcc.
Is there a specific need for 128bit bit precision? Perhaps something like QuadDouble (http://crd-legacy.lbl.gov/~dhbailey/mpdist/).<http://crd-legacy.lbl.gov/~dhbailey/mpdist/> You could also use MPFR to get arbitrary precision.
Another option I have not used, is to try clang+cuda. Recent LLVM/CLANG compilers support CUDA.
I've CC'd two developers, Christian Trott and Mark Hoemmen, that may be able to think of other options.
Hopefully you can find a solution.
James
________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Elliott, James John <jjellio at sandia.gov>
Sent: Tuesday, February 21, 2017 8:54 PM
To: Hysom, David A.; trilinos-users at trilinos.org
Subject: Re: [Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
Hi David,
Glad you got it working.... sorry it didn't work ;)
My guess, is that you are running into an issue with nvcc/gcc compatibility.
I came across:
https://svn.boost.org/trac/boost/ticket/11852
(from http://stackoverflow.com/questions/34959032/nvcc-and-boost-1-60-error-identifier-float128-is-undefined )
Which mentions that CUDA 7.5 is supposed to be able to handle float128
I do not see this 'fix' mentioned in the 7.5 release notes.
Trilinos by default will add std=c++11, so perhaps CUDA 7.5 will resolve this.
Can you possibly try CUDA 7.5?
That is my current best guess.
James
________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Hysom, David A. <hysom1 at llnl.gov>
Sent: Tuesday, February 21, 2017 3:31 PM
To: James Elliott; trilinos-users at trilinos.org
Subject: Re: [Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
Well, trilinos built; but when I attempt to compile my code I get:
/collab/usr/gapps/opnsrc/gnu/dev/lnx-2.12-o/4.8.5/include/c++/4.8.5/type_traits(269): error: identifier "__float128" is undefined
I've googled around, and it appears this is a problem with g++.
Unf, I've not found a solution.
Have you any ideas?
________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Hysom, David A. <hysom1 at llnl.gov>
Sent: Tuesday, February 21, 2017 12:36:39 PM
To: James Elliott; trilinos-users at trilinos.org
Subject: Re: [Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
James,
Success! I did as you suggested and the build went through.
Thanks for your help!
- David
________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of James Elliott <jjellio3 at ncsu.edu>
Sent: Tuesday, February 21, 2017 10:08:02 AM
To: Hysom, David A.; trilinos-users at trilinos.org
Subject: Re: [Trilinos-Users] [EXTERNAL] error building trilinos with CUDA support
Hi David,
You should not need MPI. In that case, set
CMAKE_CXX_COMPILER=/p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/bin/nvcc_wrapper
Turn MPI off.
and export NVCC_WRAPPER_DEFAULT_COMPILER=/usr/apps/gnu/4.8.5/bin/g++
Can you post the output from:
make VERBOSE=1
James
On 2/21/2017 10:57 AM, David Hysom wrote:
> James,
>
> I made these changes in my script:
>
> -D CMAKE_CXX_COMPILER:STRING="/usr/local/bin/mpiCC" \
> -D CMAKE_C_COMPILER:STRING="/usr/local/bin/mpicc" \
> -D TPL_ENABLE_MPI:BOOL=ON \
>
> (our code doesn't use mpi, which is why I had it turned off)
>
> $which mpicc
> /usr/local/bin/mpicc
>
> $mpicc --version
> gcc (GCC) 4.8.5
>
> /p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/config/nvcc_wrapper
>
> doesn't exist; I modified this to:
> export
> OMPI_CXX=/p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/bin/nvcc_wrapper
>
> I also exported the variables before running make.
>
> Unf, I'm getting the same errors as before
>
> (our code doesn't use mpi, which is why I had it turned off)
>
> Have you any other suggestions?
>
> thanks, David
>
>
> On 02/21/2017 09:06 AM, Elliott, James John wrote:
>> Hi David,
>>
>> Set
>> CMAKE_CXX_COMPILER
>> CMAKE_C_COMPILER
>>
>> To
>> mpicxx and mpicc
>>
>> This changes the compiler that mpicxx will use. Now, it will call
>> nvcc_wrapper under the hood.
>> export
>> OMPI_CXX=/p/lscratche/hysom/trilinos-12.10.1-Source/Trilinos/packages/kokkos/config/nvcc_wrapper
>>
>> This sets the compiler that NVCC_WRAPPER uses to your g++ binary
>> export NVCC_WRAPPER_DEFAULT_COMPILER=/usr/apps/gnu/4.8.5/bin/g++
>>
>> James
>>
>> ________________________________________
>> From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf
>> of David Hysom <hysom1 at llnl.gov>
>> Sent: Tuesday, February 21, 2017 9:52 AM
>> To: trilinos-users at trilinos.org
>> Subject: [EXTERNAL] [Trilinos-Users] error building trilinos with
>> CUDA support
>>
>> acording to this page:
>> https://trilinos.org/about/documentation/building-with-cuda-support/
>>
>> I should include this in my cmake script:
>>
>> -DCMAKE_CXX_FLAGS="-g -lineinfo -Xcudafe \
>> --diag_suppress=conversion_function_not_usable -Xcudafe \
>> --diag_suppress=cc_clobber_ignored -Xcudafe \
>> --diag_suppress=code_is_unreachable" \
>>
>> However, those flags are not recognized by the compiler.
>> I was able to successfully configure with cuda support
>> (see script below), however, when I then run make I get
>> these errors:
>>
>> /p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/core/src/Kokkos_Macros.hpp:163:40:
>> error: '__device__' does not name a type
>> #define KOKKOS_INLINE_FUNCTION __device__ __host__ inline
>> ^
>> /p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/core/src/impl/Kokkos_Utilities.hpp:58:1:
>> note: in expansion of macro 'KOKKOS_INLINE_FUNCTION'
>> KOKKOS_INLINE_FUNCTION
>> /p/lscratche/hysom/trilinos-12.10.1-Source/packages/kokkos/core/src/Kokkos_Core_fwd.hpp:182:3:
>> note: in expansion of macro 'KOKKOS_INLINE_FUNCTION'
>> KOKKOS_INLINE_FUNCTION static void verify(const void *) {}
>>
>> Any help will be appreciated!
>>
>> ======
>> #!/bin/bash
>>
>> . /usr/local/tools/dotkit/init.sh
>> use gcc-4.8.5p
>>
>> export
>> OMPI_CXX=/p/lscratche/hysom/trilinos-12.10.1-Source/Trilinos/packages/kokkos/config/nvcc_wrapper
>> export NVCC_WRAPPER_DEFAULT_COMPILER=/usr/apps/gnu/4.8.5/bin/g++
>> export CUDA_LAUNCH_BLOCKING=1
>>
>> export CUDA_TOOLKIT_ROOT_DIR=/opt/cudatoolkit-7.0
>> export CUDA_BIN_PATH=/opt/cudatoolkit-7.0
>>
>> TRILINOS_SOURCE_PATH=/p/lscratche/hysom/trilinos-12.10.1-Source
>> TRILINOS_INSTALL_PATH=/p/lscratchf/hysom/trilinos_cuda
>>
>> EXTRA_ARGS=$@
>>
>> rm -f CMakeCache.txt
>>
>> cmake \
>> -D CMAKE_CXX_COMPILER:STRING="/usr/apps/gnu/4.8.5/bin/g++" \
>> -D CMAKE_C_COMPILER:STRING="/usr/apps/gnu/4.8.5/bin/gcc" \
>> -DCMAKE_CXX_FLAGS="-g" \
>> -DTPL_ENABLE_CUDA=ON \
>> -DKokkos_ENABLE_Cuda=ON \
>> -DKokkos_ENABLE_Cuda_UVM=ON \
>> -D Trilinos_ENABLE_Fortran:BOOL=ON \
>> -D Trilinos_ASSERT_MISSING_PACKAGES=OFF \
>> -D Teuchos_ENABLE_COMPLEX:BOOL=OFF \
>> -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
>> -D Trilinos_ENABLE_ThreadPool:BOOL=OFF \
>> -D Trilinos_ENABLE_TEUCHOS_TIME_MONITOR:BOOL=ON \
>> -D TPL_ENABLE_Pthread:BOOL=OFF \
>> -D TPL_ENABLE_MPI:BOOL=OFF \
>> -D TPL_ENABLE_TBB:BOOL=OFF \
>> -D Trilinos_ENABLE_Kokkos:BOOL=ON \
>> -D Kokkos_ENABLE_EXAMPLES:BOOL=ON \
>> -D Kokkos_ENABLE_TESTS:BOOL=OFF \
>> -D
>> Tpetra_DefaultNode:STRING=Kokkos::Compat::KokkosCudaWrapperNode \
>> -D Trilinos_ENABLE_OpenMP:BOOL=ON \
>> -D Trilinos_ENABLE_CHECKED_STL:BOOL=OFF \
>> -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=OFF \
>> -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
>> -D Trilinos_ENABLE_Anasazi:BOOL=ON \
>> -D Trilinos_ENABLE_Tpetra:BOOL=ON \
>> -D Trilinos_ENABLE_Zoltan2:BOOL=OFF \
>> -D Trilinos_ENABLE_MueLu:BOOL=OFF \
>> -D Zoltan2_ENABLE_TESTS:BOOL=OFF \
>> -D MueLu_ENABLE_TESTS:BOOL=OFF \
>> -D Tpetra_ENABLE_TESTS:BOOL=OFF \
>> -D Trilinos_ENABLE_EXAMPLES:BOOL=OFF \
>> -D CMAKE_INSTALL_PREFIX:STRING=${TRILINOS_INSTALL_PATH} \
>> -D CMAKE_BUILD_TYPE:STRING=RELEASE \
>> -D Kokkos_ENABLE_Serial:BOOL=OFF \
>> -D DART_TESTING_TIMEOUT:STRING=600 \
>> -D Trilinos_ENABLE_Belos=ON \
>> -D Trilinos_ENABLE_Ifpack2=ON \
>> ${EXTRA_ARGS} \
>> ${TRILINOS_SOURCE_PATH}
>>
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at trilinos.org
>> https://trilinos.org/mailman/listinfo/trilinos-users
>
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at trilinos.org
> https://trilinos.org/mailman/listinfo/trilinos-users
_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at trilinos.org
https://trilinos.org/mailman/listinfo/trilinos-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20170222/2d8f5c36/attachment.html>
More information about the Trilinos-Users
mailing list