[Trilinos-Users] do{Ex,Im}port const'ness

[Baker, Christopher G.]

Still, I don't really understand what I'm doing there.

I implemented this stuff in Tpetra, and if I look away from the code for more than thirty seconds, anything except the most basic use case flees my mind.

<glances at code...>

The difficulty comes from the fact that there is more than one way to do the same thing, according to this interface. To perform the work that you just coded up, you could use any of
doImport(..., importer)
doExport(..., exporter)
doImport(..., exporter)
doExport(..., importer)
and the results would be the effectively the same, as long as the importer/exporter objects were properly created. The best approach is therefore the one which results in the clearest code. And it's best to prefer the first two over the last two, if you have a choice.

It prompts us to ask: why the ambiguity? Underlying all of these classes and methods is a single class, a low-level object in Tpetra and Epetra called Distributor. It contains a plan for efficiently distributing the data between distributed objects. Some amount of communication and organization is undertaken once and stored by the Distributor; the objects only have to know how to pack and unpack the data. At runtime, this allows a minimum number of MPI calls to be made very quickly, using pre-staged memory.

Often times, given a plan for distributing data, we need the ability to run the reverse plan. One example is sparse matvec. The plan needed to import ghost nodes for a sparse matvec is the reverse of the plan needed to distributed sparse vectors after the transpose matvec operation. Therefore, the sparse matvec operation will create a Import object (containing a Distributor) to do the matvec, and use that Import object in reverse mode when doing a transpose matvec.

doImport() with a Import object goes forward; doExport() with an Import object goes backward (uses the reverse communication plan). And mutatis mutandis for doExport(). The reason to have all four options is that we need to support the forward and reverse options with whatever objects we have on hand; an Export or an Import object. The reason to have both Export and Import is that each creates a Distributor, and we only want to create another Distributor containing the reverse plan if we have to (at runtime, by calling doImport(exporter) or doExport(importer)). That is why we prefer not to "import" with "export".

For a glance at this, look at the doImport() and doExport() methods in Tpetra_DistObject.hpp; they are each only a few lines long, and they outline a good bit of the logic that I've tried to convey here.

I don't know of a place where this functionality (identical between Epetra and Tpetra) is well-documented. I expect it is not directly used except by small percentage of users. Mike, any pointers?

Chris

cc: Mike Heroux, Trilinos-users for archival purposes