[Trilinos-Users] [EXTERNAL] Solving linear systems on GPU

[Elliott, James John]

Ashesh,

Look at the 4th template parameter for Tpetra CRS Matrix and MultiVector (the node_type).

If you built with CUDA, then

  typedef double   scalar_type;
  typedef int local_ordinal_type;
  typedef int global_ordinal_type;
  // Use the CUDA execution space.
  // A 'device' as a tuple of ExecutionSpace, MemorySpace
  // Specify the ExecSpace, but leave the other out, and it will use the default memory space for that execution space (CUDA UVM)
  typedef Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::CUDA> node_type;

  // Now define your matrix, MV, etc..
  typedef Tpetra::CrsMatrix<scalar_type, local_ordinal_type, global_ordinal_type, node_type> crs_matrix_type;

The device is a Kokkos device, but wrapped by Tpetra for compatibility reasons.
https://trilinos.org/docs/dev/packages/kokkos/doc/html/structKokkos_1_1Device.html

James



________________________________
From: Trilinos-Users <trilinos-users-bounces at trilinos.org> on behalf of Rajamanickam, Sivasankaran <srajama at sandia.gov>
Sent: Tuesday, January 31, 2017 8:38 AM
To: Ashesh Chattopadhyay
Cc: Trilinos-Users at trilinos.org
Subject: Re: [Trilinos-Users] [EXTERNAL] Solving linear systems on GPU


Ashes

   The execution space is based on the Tpetra matrix and vector execution space.


    Let me see if we can find a short example for you. BTW, please CC the list so you get responses from different people and the responses are archived.


-Siva


________________________________
From: Ashesh Chattopadhyay <ashesh6810 at gmail.com>
Sent: Tuesday, January 31, 2017 8:18 AM
To: Rajamanickam, Sivasankaran
Subject: Re: [Trilinos-Users] [EXTERNAL] Solving linear systems on GPU

Sir,
I was using this function to set everything up in Belos to solve the linear system.

template<class MV, class OP>
MV  solve ( MV& X, const MV& B, const OP& A)
{
  using Teuchos::ParameterList;
  using Teuchos::parameterList;
  using Teuchos::RCP;
  using Teuchos::rcp;
  using Teuchos::rcpFromRef;
  typedef typename MV::scalar_type scalar_type;

  RCP<ParameterList> solverParams = parameterList();
solverParams->set ("Num Blocks", 40);
solverParams->set ("Maximum Iterations", 5000);
solverParams->set ("Convergence Tolerance", 1.0e-3);
Belos::SolverFactory<scalar_type, MV, OP> factory;
  RCP<Belos::SolverManager<scalar_type, MV, OP> > solver =
    factory.create ("GMRES", solverParams);


  typedef Belos::LinearProblem<scalar_type, MV, OP> problem_type;
  RCP<problem_type> problem =
    rcp (new problem_type (rcpFromRef (A), rcpFromRef (X), rcpFromRef (B)));
 problem->setProblem ();

  solver->setProblem (problem);
  Belos::ReturnType result = solver->solve();
 int numIters = solver->getNumIters();
if (result == Belos::Converged) {
   std::cout << "The Belos solve took " << numIters << " iteration(s) to reach "
      "a relative residual tolerance of " << 1.0e-4 << "." << std::endl;
  } else {
   std::cout << "The Belos solve took " << numIters << " iteration(s), but did not reach "
      "a relative residual tolerance of " << 1.0e-4 << "." << std::endl;
  }


return X;

Is there a way to specify, something like execution space as device or something like that, the way one can do in Kokkos to execute on GPU? Or any other way, kindly point me to a reference from which I can learn how to do it.

Thanks
Ashesh



On Tue, Jan 31, 2017 at 8:06 AM, Ashesh Chattopadhyay <ashesh6810 at gmail.com<mailto:ashesh6810 at gmail.com>> wrote:

Sir,
Thank you. Yes , I have a Tpetra Crs Matrix A and a multivector b. I simply wanna solve Ax=b,without preconditioning. My A matrix is a saddle point matrix coming out of a modified nodal analysis algorithm. Can you please point me to an example or documentation as to how to use Belos to solve this on GPU? Till now I was doing this on CPU using GMRES in belos, the way the examples are there on git.

Thanks
Ashesh

On Jan 31, 2017 7:33 AM, "Rajamanickam, Sivasankaran" <srajama at sandia.gov<mailto:srajama at sandia.gov>> wrote:
...



--
Research Assistant, Multi-Scale Multi-Physics Computation Lab
University of Texas at El Paso
Texas, El Paso, USA
(+1) 915-355-5013
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://trilinos.org/pipermail/trilinos-users/attachments/20170131/cc7efb93/attachment.html>