[Trilinos-Users] [EXTERNAL] Issues with parallelization of EpetraVectors, compatibility with NOX

[Heroux, Michael A]

Debashis,

Is your variable A the degrees of freedom at a single mesh point and then B the DOFs at the next grid point, so you are tracking 10 variables per grid point?  If so, then how many grid points do you typically have?  Many people have done coupled PDEs with Trilinos.  I am sure we can help once we make sure we understand your situation accurately.

Here are a couple of things to consider in the mean time:

  *   If you interleave your variables (A0, B0, A1, B1, …, A9, B9), you could get the desired effect, but it sounds like you have another ordering that is needed for other computations.
  *   You could use a block map (Epetra_BlockMap) to represent your data.  You can set the block size to 10 when you define the map.  This will force all ten components associated with a grid point to remain together, although this might not be exactly what you want either.

Also, starting with MPI only is the best approach for now.  Using OpenMP can be viewed as an optimization step for later.

Mike

From: <Kar>, Debashis * <Debashis.Kar at fda.hhs.gov<mailto:Debashis.Kar at fda.hhs.gov>>
Date: Friday, August 24, 2012 3:03 PM
To: "trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>" <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov>>
Subject: [EXTERNAL] [Trilinos-Users] Issues with parallelization of EpetraVectors, compatibility with NOX

Hi all

I wished to use a vector like data structure while solving a system of coupled nonlinear equations (PDEs). I am using the LinearSystemAztec00 solver of the NOX package and Epetra Vectors as the data structure. The independent variables are made part of the same vector U - such that say we need 10 entries for each of these variables, then the first 10 entries of U belong to variable A, the next 10 to variable B and so on.

We intended to parallelize the computation so as to speed things up. The issue we are facing is this - say we divide EpetraVector U between 2 machines; then instead of the first 5 entries of each A and B being fed to machine 1 and the next 5 (of A, B) fed to machine 2, what happens is that the first 10 entries of U (all of A) are fed to machine 1 and the next 10 (all of B) are fed to machine 2.

One way to overcome this might be to use a MultiVector instead where each independent variable will correspond to a row in the U (such that A=U(1,;), B = U(2,:) and so on). Is MultiVector compatible with the NOX solvers? If not can you point out to relevant data structures which are compatible with NOX and can be effectively parallelized? Otherwise we might need to redefine the mapping between EpetraVector U and the set of independent variables (A,B,…) each time we vary the number of machines during a parallel run, which is cumbersome and non-optimal.

Also for parallelization is it advisable to use OpenMP or should one only stick to MPI ?

Best
Debashis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://software.sandia.gov/pipermail/trilinos-users/attachments/20120824/31454509/attachment.html