[Trilinos-Users] Tpetra/Kokkos Ghosting (Duncan Karnitz)

[Hoemmen, Mark]

Hi Duncan!  I’m not sure why I’m only getting these messages on a three-week delay.  Anyway, I’ll do my best to help out.

On 3/21/17, 6:11 PM, "Trilinos-Users on behalf of trilinos-users-request at trilinos.org" <trilinos-users-bounces at trilinos.org on behalf of trilinos-users-request at trilinos.org> wrote:
>Date: Mon, 6 Mar 2017 14:08:13 -0500 (EST)
>From: Duncan Karnitz <dlk at thermoanalytics.com>
>To: trilinos-users at trilinos.org
>Subject: [Trilinos-Users] Tpetra/Kokkos Ghosting
>
>I'm working on learning the Tpetra/Kokkos stack by attempting to program a 1D upwind scheme for the linear Burger's equation. So far, I've been able to use Tpetra::Vector and Kokkos::parallel_for() to explicitly time step the solution with shared memory parallel. However, I'm having trouble understanding how one would extend to distributed memory parallel where the spatial dependencies may cross process boundaries.
>
>I'm familiar with how one might synchronize ghost nodes manually with MPI and C/C++ vectors. But I am wondering if there are ways to do this easily within the Tpetra/Kokkos/Teuchos stack.

Tpetra can help you do this with very few lines of code.  Tpetra Lessons 02 and 05 may help:

https://trilinos.org/docs/dev/packages/tpetra/doc/html/Tpetra_Lesson02.html 
https://trilinos.org/docs/dev/packages/tpetra/doc/html/Tpetra_Lesson05.html

Boundary exchange is just a global sparse all-reduce.  Tpetra::Export implements this operation.  Suppose that you can partition your degrees of freedom (DOFs) on each MPI process into two categories:

1. OWNED: the MPI process is responsible for computing these DOFs
2. SHARED: the MPI process contributes to the equations for computing these DOFs

Create two Maps: an owned Map, and a shared Map.  Then create two corresponding Vectors.  If you do an Export from the shared Vector to the owned Vector, with ADD combine mode, that will += all the remote contributions into each process’ entries.  Be sure to reuse the Export, since setting it up is expensive (constructing a boundary exchange, vs. applying it).

If you would like more details, both the Tpetra and Epetra documentation can help out.  Also feel free to ask questions.  Please send me e-mail or post to GitHub, since trilinos-users seems to have a delay.

mfh