[Trilinos-Users] [EXTERNAL] Building Trilinos with CUDA support

[Phipps, Eric T]

Hi Max,

You need to use nvcc_wrapper to build Trilinos with Cuda support.
Unfortunately this currently isn¹t well documented.  The nvcc_wrapper
script is distributed inside Kokkos within Trilinos.  A challenge is
enabling both Cuda and MPI.  I do this using OpenMPI where I can redefine
the underlying compiler via an environment variable.  For example, I set
these variables:

export 
OMPI_CXX=/home/etphipp/Trilinos/Trilinos/packages/kokkos/config/nvcc_wrappe
r
export 
NVCC_WRAPPER_DEFAULT_COMPILER=/usr/local/software/gcc/4.8.2/bin/g++-4.8
export CUDA_LAUNCH_BLOCKING=1


This tells the mpicxx to use nvcc_wrapper as the underlying compiler, and
the second variable tells nvcc_wrapper which host compiler I want to use
(note:  I think it is still true that nvcc fails when using a host
compiler newer than 4.8.x due to some incorrect parsing of some the
standard library headers).  The last line is for UVM (which the Tpetra
stack in Trilinos currently requires) and forces a fence after each kernel
launch.  If you can¹t redefine the compiler used by MPI, you will probably
have to either forego the use of the MPI compiler wrappers and set the MPI
includes/libraries directly, or recompile your MPI library with
nvcc_wrapper (if possible).

Below is a CMake configure script fragment to then configure Trilinos:

-D 
CMAKE_CXX_COMPILER=/usr/local/software/openmpi/1.8.1-gcc4.8.2/bin/mpicxx \
 -D CMAKE_C_COMPILER=/usr/local/software/openmpi/1.8.1-gcc4.8.2/bin/mpicc \
 -D 
CMAKE_Fortran_COMPILER=/usr/local/software/openmpi/1.8.1-gcc4.8.2/bin/mpif7
7 \
 -D CMAKE_CXX_FLAGS="-g -lineinfo -Xcudafe
--diag_suppress=conversion_function_not_usable -Xcudafe
--diag_suppress=cc_clobber_ignored -Xcudafe
--diag_suppress=code_is_unreachable" \
-D TPL_ENABLE_MPI=ON \
-D TPL_ENABLE_CUDA=ON \
-D Kokkos_ENABLE_Cuda=ON \
 -D Kokkos_ENABLE_Cuda_UVM=ON \





The CMAKE_CXX_FLAGS line adds some nvcc_wrapper command-line arguments to
disable some superfluous warnings generated by nvcc.

Note that this process generally works when compiling Kokkos, Tpetra, and
the Tpetra-based solvers in Trilinos.  However there are packages in
Trilinos that nvcc fails to compile due to bugs in nvcc.  Things are
getting better as Nvidia fixes nvcc, but your mileage may vary.

Hope that helps.

-Eric

On 10/31/15, 8:29 PM, "Trilinos-Users on behalf of Max Grossman"
<trilinos-users-bounces at trilinos.org on behalf of jmg3 at rice.edu> wrote:

>Hi all,
>
>I¹m a new Trilinos user (primarily the Tpetra and Kokkos packages so
>far), and I could use some help with the Trilinos build system.
>
>I¹ve successfully built and tested Trilinos with MPI and OpenMP support,
>but now I¹d like to try out some Tpetra operations on NVIDIA GPUs so I¹m
>trying to compile with CUDA support. I run the Trilinos cmake with the
>following options:
>
>-DTPL_ENABLE_CUDA=ON
>-DTPL_ENABLE_Thrust=ON
>-D MPI_C_COMPILER=mpiicc
>-D MPI_CXX_COMPILER=mpiicpc
>-D MPI_Fortran_COMPILER=mpiifort
>-D Kokkos_ENABLE_Cuda=ON
>-D Kokkos_ENABLE_Cuda_UVM=ON
>-D 
>Thrust_INCLUDE_DIRS=/opt/apps/software/Compiler/GCC/4.4.7/CUDA/6.5.14/incl
>ude
>-D Trilinos_ENABLE_DEVELOPMENT_MODE:BOOL=OFF
>-D Trilinos_ENABLE_OpenMP=ON
>-D Tpetra_INST_OPENMP:BOOL=ON
>-D Tpetra_INST_SERIAL=ON
>
>and when I then try to build with make, I get a number of error messages
>because files referencing CUDA objects aren¹t being built by nvcc, such
>as:
>
>In file included from
>/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_CudaSpac
>e.hpp:60:0,
>                 from
>/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_Cuda.hpp
>:56,
>                 from
>/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_Core.hpp
>:53,
>                 from
>/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/impl/Kokkos_Cor
>e.cpp:44:
>/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Cuda/Kokkos_Cud
>a_BasicAllocators.hpp:82:22: error: ŒcudaChannelFormatDesc¹ has not been
>declared
>                    , cudaChannelFormatDesc const & desc
>
>This makes sense to me because I¹ve set MPI_C_COMPILER and
>MPI_CXX_COMPILER to non-nvcc compilers, but Trilinos also doesn¹t seem to
>support compilation with nvcc (errors about unsupported flags during
>cmake). Looking at the Kokkos package github, it seems like there¹s a
>nvcc_wrapper tool that might be helpful but which doesn¹t seem to come
>packaged with Trilinos, so I¹m also having trouble understanding why the
>files in the Kokkos github are different from the files included under
>packages/kokkos/ in the Trilinos download.
>
>So, anyone know what I might be doing wrong, or missing? Should I in fact
>be using nvcc? Is there a cmake flag I¹m missing? Or is the nvcc_wrapper
>tool the answer?
>
>Thanks for any help!
>
>Max
>_______________________________________________
>Trilinos-Users mailing list
>Trilinos-Users at trilinos.org
>https://trilinos.org/mailman/listinfo/trilinos-users