[Trilinos-Users] Epetra_Vector with overlap

Heroux, Michael A maherou at sandia.gov
Thu Jul 8 08:10:38 MDT 2010


Ross,

Yes, it would for the typical usage (although inf-norm would still be
accurate).  In general, collectives are valid only on vectors with a
one-to-one map (easily checked by looking at the bool
vector.Map().UniqueGIDs()) but we don't prohibit use of the collectives on
other vectors because there are situations where they can be helpful in
advanced usages, e.g., a RAID distribution of the data,where collectives can
be used as check-sums.

The bottom line is that, typical usage of vectors with off-processor
elements should not include collective operations.

Mike


On 7/8/10 8:51 AM, "Roscoe Bartlett" <rabartl at sandia.gov> wrote:

> If a vector contains duplicated off-processor elements does that not mess up
> dot product and norm calculations?
> 
> - Ross
> 
> 
>> -----Original Message-----
>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
>> users-bounces at software.sandia.gov] On Behalf Of Pawlowski, Roger P
>> Sent: Wednesday, July 07, 2010 5:58 PM
>> To: Heroux, Michael A; Salinger, Andrew; Berndt, Markus (LANL)
>> Cc: trilinos-users at software.sandia.gov
>> Subject: Re: [Trilinos-Users] Epetra_Vector with overlap
>> 
>> Option 2 should not be fragile for NOX algorithms except in one single
>> case.  The generic NOX algorithms have no notion of an overlapped
>> vector or that we are even running in parallel (except for outputting
>> text to cout).  It only uses unique-GIDs map/graphs.  NOX passes back
>> unique GID vectors to the user interface.   The gather/scatters with
>> overlapped objects occur in the user implemented interface for computeF
>> and computeJacobian).  The only place where overlapping might be an
>> issue is for the finite difference coloring algorithm that is specific
>> to the Epetra support.
>> 
>> We should not be initializing nox solvers with overlapped vectors since
>> the size() method that is used in scaling some stopping criteria
>> tolerances will yield different accuracies when running with different
>> numbers of processors.
>> 
>> Roger
>> ________________________________________
>> From: trilinos-users-bounces at software.sandia.gov [trilinos-users-
>> bounces at software.sandia.gov] On Behalf Of Heroux, Michael A
>> [maherou at sandia.gov]
>> Sent: Wednesday, July 07, 2010 3:10 PM
>> To: Salinger, Andrew; Berndt, Markus (LANL)
>> Cc: trilinos-users at software.sandia.gov
>> Subject: Re: [Trilinos-Users] Epetra_Vector with overlap
>> 
>> Andy,
>> 
>> I think careful use of Option 2 is OK.  It is used quite frequently for
>> what
>> Markus is doing.
>> 
>> Mike
>> 
>> 
>> On 7/7/10 3:25 PM, "Andrew Salinger" <agsalin at sandia.gov> wrote:
>> 
>>> 
>>> Marcus,
>>> 
>>> I see two choices:
>>> In both cases you have 2 Maps: owned (unique GIDs) and overlap.
>>> (1) Have two copies of the vector (which you want to avoid)
>>> (2) Construct your overlap map so that the owned elements
>>> are contiguous. Then your owned vector can be constructed
>>> as a View inside the overlap vector.
>>> In your example, if the overlap map on PE1 is
>>>  4 5 6 7 8 9   or    5 6 7 8 9 4,
>>> then the owned vector can be constructed with
>>> the map  5 6 7 8 9 and a pointer to the element
>>> with "5" in the overlap map.
>>> 
>>> I think (1) is the Best Practices approach. Option (2)
>>> is fragile, since an algorithm (say in NOX) that makes a
>>> copy of an owned vector will allocate it at the shorter
>>> length. If you try this trick on that vector, it will
>>> try and write past the allocated length.
>>> 
>>> Andy
>>> 
>>> -----Original Message-----
>>> From: trilinos-users-bounces at software.sandia.gov
>>> [mailto:trilinos-users-bounces at software.sandia.gov] On Behalf Of
>> Markus Berndt
>>> Sent: Wednesday, July 07, 2010 9:46 AM
>>> To: trilinos-users at software.sandia.gov
>>> Subject: [Trilinos-Users] Epetra_Vector with overlap
>>> 
>>> Is it possible to set up communication schedules for an Epetra_Vector
>> with
>>> overlap that can be used to simply update the overlap region in
>> place? The
>>> documentation for Epetra_Export and Epetra_Import indicates that one
>> of the
>>> maps that these use to initialize communication patterns must have
>> GIDs that
>>> are unique. I interpret that as one of the vectors in an Import or
>> Export call
>>> must have no overlap at all.
>>> 
>>> 
>>> To illustrate, here is an example:
>>> 
>>> x_ovl is an Epetra_Vector with overlap such that on PE0 it owns GIDs
>> 0 1 2 3 4
>>> and has a copy of 5 and on PE1 it owns GIDs 5 6 7 8 9 and has a copy
>> of 4.
>>> 
>>> How do I set up a communication pattern that
>>> 
>>> - sends x_ovl(5) from PE1 to PE0 and
>>> - sends x_ovl(4) from PE0 to PE1
>>> 
>>> without an intermediate vector that has only the local data and no
>> overlap at
>>> all.
>>> 
>>> If this is not possible, what would you consider a best practices
>> approach to
>>> working with vectors that have overlap that avoids unnecessary copy
>> operations
>>> between equivalent vectors with and without overlap?
>>> 
>>> Thanks
>>> 
>>> - Markus
>>> 
>>> --
>>> Markus Berndt, LANL CCS-2, Mail Stop D413, 505-665-4711
>>> 
>>> 
>>> _______________________________________________
>>> Trilinos-Users mailing list
>>> Trilinos-Users at software.sandia.gov
>>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>>> 
>>> _______________________________________________
>>> Trilinos-Users mailing list
>>> Trilinos-Users at software.sandia.gov
>>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>> 
>> 
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at software.sandia.gov
>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>> 
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at software.sandia.gov
>> http://software.sandia.gov/mailman/listinfo/trilinos-users
> 
> _______________________________________________
> Trilinos-Users mailing list
> Trilinos-Users at software.sandia.gov
> http://software.sandia.gov/mailman/listinfo/trilinos-users




More information about the Trilinos-Users mailing list