[Trilinos-Users] Copying Tpetra vectors and matrices between host and GPU

[ct37]

Hello,

I am currently experimenting with Tpetra's CUDA backend. As a first  
test, I would like to perform some computations and eventually solve a  
linear system using the GPU, starting with vectors and matrices that  
were assembled in host memory. However, once I attempt to copy a  
Tpetra::Vector from the host to the GPU, I get dozens of errors, all  
with the message

   Assertion `Kokkos::View ERROR: attempt to access inaccessible  
memory space` failed.

My code looks as follows:

   // Initialization
   Kokkos::OpenMP::initialize(threads, 1);
   Kokkos::Cuda::initialize(Kokkos::Cuda::SelectDevice(0));

   // Type definitions
   typedef Kokkos::Compat::KokkosOpenMPWrapperNode HOST_Node;
   typedef Tpetra::Map<int, int, HOST_Node> HOST_Map;
   typedef Tpetra::Vector<double, int, int, HOST_Node> HOST_Vector;
   typedef Kokkos::Compat::KokkosCudaWrapperNode CUDA_Node;
   typedef Tpetra::Map<int, int, CUDA_Node> CUDA_Map;
   typedef Tpetra::Vector<double, int, int, CUDA_Node> CUDA_Vector;

   // Create vectors.
   const int n = 1024;
   auto comm = Teuchos::DefaultComm<int>::getComm();
   auto map_host = Teuchos::rcp(new HOST_Map(n, 0, comm));
   auto v_host = Teuchos::rcp(new HOST_Vector(map_host));
   auto map_cuda = Teuchos::rcp(new CUDA_Map(n, 0, comm));
   auto v_cuda = Teuchos::rcp(new CUDA_Vector(map_cuda));

   // Initialize v_host.
   // ...

   double result_host, result_cuda;
   // Test computation on host.
   result_host = v_host->dot(*v_host);  // Works!
   // Test computation on GPU.
   v_cuda->putScalar(1.0);
   result_cuda = v_cuda->(*v_cuda);     // Works!
   // Now copy data from host.
   Tpetra::deep_copy(*v_cuda, *v_host); // Error!
   result_cuda = v_cuda->(*v_cuda);

Is Tpetra::deep_copy() the correct function to be used here? Also, how  
would I copy a Tpetra::CrsMatrix? All code examples that I was able to  
find use only one node type, but not the OpenMP and CUDA backends  
simultaneously.

Best regards,
Christopher