[Trilinos-Users] creating Tpetra maps from off-processor data

[Templeton, Jeremy Alan]

Thanks Mark!  We’ll give it a try.
Jeremy

On Nov 5, 2014, at 11:14 AM, Hoemmen, Mark <mhoemme at sandia.gov<mailto:mhoemme at sandia.gov>> wrote:

On 11/5/14, 11:42 AM, "Templeton, Jeremy Alan" <jatempl at sandia.gov<mailto:jatempl at sandia.gov>> wrote:
A simple example of the case we are considering is that process 1 owns a global id *and* knows that process 0 needs access to that global id’s data.  However, process 0 doesn’t know anything about that global id because it doesn’t own it.

Thanks for the clarification!  I think we can help you.  You could try using Tpetra::Distributor (see Tpetra_Distributor.hpp).  We use Distributor to implement communication pattern setup and execution in Import and Export.  It's not so pretty because it's meant less for users, but it might help you here.

In your case, you know that Process 1 wants to send something to Process 0.  In general, each process p knows that it wants to send something to zero or more other processes q_1, q_2, ….  This suggests using createFromSends to set up the Distributor:

using Teuchos::Comm;
using Teuchos::RCP;
typedef Teuchos::Array<int>::size_type size_type;

RCP<const Comm<int> > comm = …; // your communicator
Tpetra::Distributor d (comm);
// Processes to which my process will send.
Teuchos::Array<int> outgoingProcs = …;
// createFromSends returns the number of
// incoming messages on the calling process.
const size_t myNumRecvs = d.createFromSends (outgoingProcs ());

Now the Distributor is set up for communication.  Next, you actually need to have it send the data.  Distributor can do this for you!  Each message in this case consists of a set of global indices which belong to the target process.  One issue is that each receiving process has to know how many messages it will receive.  You can actually use the Distributor we constructed above to send this information.  Before sending the global indices, send a count of global indices first.

const size_type numOutgoing = outgoingProcs.size ();
Teuchos::Array<size_t> outgoingCounts (numOutgoing);
for (size_type k = 0; k < numOutgoing; ++k) {
  // number of global indices to send to outgoingProcs[k]
  outgoingCounts[k] = …;
}

// The Distributor has already told you how many
// incoming messages to expect.
Teuchos::Array<size_t> incomingCounts (myNumRecvs);
// Communicate to get message sizes.
// This is the 3-argument version of doPostsAndWaits.
d.doPostsAndWaits<size_t> (outgoingCounts (), numOutgoing, incomingCounts ());

Now each process knows how many global indices it will receive from each other process that will send it a message.

You may now use the 4-argument version of doPostsAndWaits to communicate the actual global indices.  You have to pack all outgoing messages contiguously, and incoming messages come in similarly packed.

// Sum of all entries of outgoingCounts.
const size_t totalNumOutgoingIndices = …;
Teuchos::Array<GO> exports (totalNumOutgoingIndices);

size_t curOffset = 0;
for (size_type k = 0; k < numOutgoing; ++k) {
  const size_t curNumInds = outgoingCounts[k];
  // Fill exports[curOffset, curOffset + curNumInds - 1] with
  // outgoing indices for process outgoingProcs[k].
  for (size_type j = 0; j < curNumInds; ++j) {
    exports[j + curOffset] = …;
  }
  curOffset += curNumInds;
}

// Sum of all entries of incomingCounts.
const size_t totalNumIncomingIndices = …;
Teuchos::Array<GO> imports (totalNumIncomingIndices);

// Communicate the global indices!
d.doPostsAndWaits<GO> (exports (), outgoingCounts (), imports (), incomingCounts ());

I've left unpacking the 'imports' array as an exercise to the reader ;-)

Hope this helps!
mfh


--------------------------------------------------------
Jeremy A. Templeton, Ph.D.
Thermal/Fluid Sciences & Engineering
jatempl at sandia.gov<mailto:jatempl at sandia.gov>
http://tiny.sandia.gov/jatempl
925-294-1429





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20141105/3f73ee13/attachment.html>