[Trilinos-Users] Building Trilinos with CUDA support

[Max Grossman]

Hi all,

I’m a new Trilinos user (primarily the Tpetra and Kokkos packages so far), and I could use some help with the Trilinos build system.

I’ve successfully built and tested Trilinos with MPI and OpenMP support, but now I’d like to try out some Tpetra operations on NVIDIA GPUs so I’m trying to compile with CUDA support. I run the Trilinos cmake with the following options:

-DTPL_ENABLE_CUDA=ON
-DTPL_ENABLE_Thrust=ON
-D MPI_C_COMPILER=mpiicc
-D MPI_CXX_COMPILER=mpiicpc
-D MPI_Fortran_COMPILER=mpiifort
-D Kokkos_ENABLE_Cuda=ON
-D Kokkos_ENABLE_Cuda_UVM=ON
-D Thrust_INCLUDE_DIRS=/opt/apps/software/Compiler/GCC/4.4.7/CUDA/6.5.14/include
-D Trilinos_ENABLE_DEVELOPMENT_MODE:BOOL=OFF
-D Trilinos_ENABLE_OpenMP=ON
-D Tpetra_INST_OPENMP:BOOL=ON
-D Tpetra_INST_SERIAL=ON

and when I then try to build with make, I get a number of error messages because files referencing CUDA objects aren’t being built by nvcc, such as:

In file included from /home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_CudaSpace.hpp:60:0,
                 from /home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_Cuda.hpp:56,
                 from /home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Kokkos_Core.hpp:53,
                 from /home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/impl/Kokkos_Core.cpp:44:
/home/jmg3/trilinos-12.2.1-Source/packages/kokkos/core/src/Cuda/Kokkos_Cuda_BasicAllocators.hpp:82:22: error: ‘cudaChannelFormatDesc’ has not been declared
                    , cudaChannelFormatDesc const & desc

This makes sense to me because I’ve set MPI_C_COMPILER and MPI_CXX_COMPILER to non-nvcc compilers, but Trilinos also doesn’t seem to support compilation with nvcc (errors about unsupported flags during cmake). Looking at the Kokkos package github, it seems like there’s a nvcc_wrapper tool that might be helpful but which doesn’t seem to come packaged with Trilinos, so I’m also having trouble understanding why the files in the Kokkos github are different from the files included under packages/kokkos/ in the Trilinos download.

So, anyone know what I might be doing wrong, or missing? Should I in fact be using nvcc? Is there a cmake flag I’m missing? Or is the nvcc_wrapper tool the answer?

Thanks for any help!

Max