[Trilinos-Users] coupling Kokkos and Trilinos

[BLOCH Helene]

Hi, 

We are using Kokkos and Trilinos, especially the packages Tpetra, Belos and MueLu, and we have encountered some difficulties to update the matrix on the device.

 We are currently trying to couple Trilinos with an existing code parallelized with Kokkos and MPI. This code implements an explicit finite volume scheme on a fixed Cartesian grid, with an MPI domain decomposition with ghost cells. We are trying to add an implicit solver, for the finite volume scheme, using backward Euler time integration. We are solving a sparse matrix, using Tpetra, with a linear solver from Belos, with an AMG preconditioner from MueLu. At each iteration of a time loop, we have to
- call the explicit solver
- update the matrix with data coming from the explicit solver
- solve the linear system
- update the solution of the explicit solver with the solution of the implicit solver
As the data coming from the explicit solver lies on the device, it would be great to update the matrix on the device, using methods from KokkosSparse::CrsMatrix. Because we already have a domain decomposition with ghost cells, the matrix has a row map and a column map that are not the same. A first test is performed on the 2D Poisson equation, MueLu gives the following error:
Throw test that evaluated to true: !goodMap
MueLu: TentativePFactory_kokkos: for now works only with good maps (i.e. "matching" row and column maps)
Here are our questions:
- did we miss some examples on the Trilinos wiki to update the matrix on the device ?
- assuming the only way to deal with the domain decomposition is to use two different maps, is there a way to still use MueLu as a preconditioner ?

Thank you for your help

Hélène Bloch