[Trilinos-Users] [EXTERNAL] Solving linear systems on GPU

Ashesh Chattopadhyay ashesh6810 at gmail.com
Tue Jan 31 12:55:19 EST 2017

Thanks. This helps a lot. I was looking into the node type and I guess I
understand it now. Once again, thank you.


Look at the 4th template parameter for Tpetra CRS Matrix and MultiVector
(the node_type).

If you built with CUDA, then

  typedef double   scalar_type;
  typedef int local_ordinal_type;
  typedef int global_ordinal_type;
  // Use the CUDA execution space.
  // A 'device' as a tuple of ExecutionSpace, MemorySpace
  // Specify the ExecSpace, but leave the other out, and it will use the
default memory space for that execution space (CUDA UVM)
  typedef Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::CUDA> node_type;

  // Now define your matrix, MV, etc..
  typedef Tpetra::CrsMatrix<scalar_type, local_ordinal_type,
global_ordinal_type, node_type> crs_matrix_type;

The device is a Kokkos device, but wrapped by Tpetra for compatibility


*From:* Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of
Rajamanickam, Sivasankaran <srajama at sandia.gov>
*Sent:* Tuesday, January 31, 2017 8:38 AM
*To:* Ashesh Chattopadhyay
*Cc:* Trilinos-Users at trilinos.org
*Subject:* Re: [Trilinos-Users] [EXTERNAL] Solving linear systems on GPU


   The execution space is based on the Tpetra matrix and vector execution

    Let me see if we can find a short example for you. BTW, please CC the
list so you get responses from different people and the responses are


*From:* Ashesh Chattopadhyay <ashesh6810 at gmail.com>
*Sent:* Tuesday, January 31, 2017 8:18 AM
*To:* Rajamanickam, Sivasankaran
*Subject:* Re: [Trilinos-Users] [EXTERNAL] Solving linear systems on GPU

I was using this function to set everything up in Belos to solve the linear

template<class MV, class OP>
MV  solve ( MV& X, const MV& B, const OP& A)
  using Teuchos::ParameterList;
  using Teuchos::parameterList;
  using Teuchos::RCP;
  using Teuchos::rcp;
  using Teuchos::rcpFromRef;
  typedef typename MV::scalar_type scalar_type;

  RCP<ParameterList> solverParams = parameterList();
solverParams->set ("Num Blocks", 40);
solverParams->set ("Maximum Iterations", 5000);
solverParams->set ("Convergence Tolerance", 1.0e-3);
Belos::SolverFactory<scalar_type, MV, OP> factory;
  RCP<Belos::SolverManager<scalar_type, MV, OP> > solver =
    factory.create ("GMRES", solverParams);

  typedef Belos::LinearProblem<scalar_type, MV, OP> problem_type;
  RCP<problem_type> problem =
    rcp (new problem_type (rcpFromRef (A), rcpFromRef (X), rcpFromRef (B)));
 problem->setProblem ();

  solver->setProblem (problem);
  Belos::ReturnType result = solver->solve();
 int numIters = solver->getNumIters();
if (result == Belos::Converged) {
   std::cout << "The Belos solve took " << numIters << " iteration(s) to
reach "
      "a relative residual tolerance of " << 1.0e-4 << "." << std::endl;
  } else {
   std::cout << "The Belos solve took " << numIters << " iteration(s), but
did not reach "
      "a relative residual tolerance of " << 1.0e-4 << "." << std::endl;

return X;

Is there a way to specify, something like execution space as device or
something like that, the way one can do in Kokkos to execute on GPU? Or any
other way, kindly point me to a reference from which I can learn how to do


On Tue, Jan 31, 2017 at 8:06 AM, Ashesh Chattopadhyay <ashesh6810 at gmail.com>

> Sir,
> Thank you. Yes , I have a Tpetra Crs Matrix A and a multivector b. I
> simply wanna solve Ax=b,without preconditioning. My A matrix is a saddle
> point matrix coming out of a modified nodal analysis algorithm. Can you
> please point me to an example or documentation as to how to use Belos to
> solve this on GPU? Till now I was doing this on CPU using GMRES in belos,
> the way the examples are there on git.
> Thanks
> Ashesh
> On Jan 31, 2017 7:33 AM, "Rajamanickam, Sivasankaran" <srajama at sandia.gov>
> wrote:
>> ...

Research Assistant, Multi-Scale Multi-Physics Computation Lab
University of Texas at El Paso
Texas, El Paso, USA
(+1) 915-355-5013 <(915)%20355-5013>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20170131/68f665d5/attachment.html>

More information about the Trilinos-Users mailing list