[Trilinos-Users] Performance Question about Epetra_FECrsMatrix (and probably Tpetra::CrsMatrix too)

[Baker, Christopher G.]

Pat,

The LID() lookup will happen regardless of the storage, because the row
index passed to SumIntoGlobalValues is a global row (regardless of whether
it is locally owned or not.)

Currently, this method in Epetra_BlockMap is not inlined; neither is it
inlined in Tpetra::Map. Perhaps it should be? That could speedup the
computation (though measurement would be coarse, as its timing will be
absorbed into SumIntoGlobalValues.)

It is possible that this would be slower in Tpetra than Epetra, as
Epetra_BlockMap goes through some hurdles to make this operation fast;
Tpetra has not implemented all of these yet.

Ultimately, there are two things that have to be done: determine (based on
owned or not-owned) where/how to store the indices, and then storing the
indices (which may entail converting the global row index to a local row).
For non-trivial (i.e., non-contiguous) maps, even doing the latter
requires non-trivial computational overhead.

Chris


On 3/31/11 9:32 AM, "Williams, Alan B" <william at sandia.gov> wrote:

>Pat,
>I haven't done any recent profiling of the FECrsMatrix fill operations.
>How high are the LID operations in your profile, compared to the fill in
>general and your coefficient-calculation code, etc.?
>
>I believe the same issue occurs in Tpetra, since the matrix has to decide
>whether each contribution belongs on the local processor or not.
>
>Perhaps a significant change is necessary to address this. One
>possibility would be to have the matrix not check local-vs-global for
>each row but instead store all contributions locally and only distinguish
>between local and non-local at the global-assemble step.
>
>Alan
>
>
>> -----Original Message-----
>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
>> users-bounces at software.sandia.gov] On Behalf Of Pat Notz
>> Sent: Wednesday, March 30, 2011 12:26 PM
>> To: trilinos-users at software.sandia.gov
>> Subject: [Trilinos-Users] Performance Question about Epetra_FECrsMatrix
>> (and probably Tpetra::CrsMatrix too)
>> 
>> I'm doing typical FEM assemblies using an Epetra_FECrsMatrix and it
>> appears to be working correctly. However, I've been profiling the
>> serial performance and I have some concerns.
>> 
>> In general, we're making off-processor contributions so we have to use
>> global row/col ids. The SumIntoGlobalValues method does some
>> gymnastics to see if the contributions are local or global and that
>> results in a lot of calls to LID(gid) -- it's LID() that's showing up
>> high in my profile.
>> 
>> As far as I can tell, it appears that you can use local ids for
>> locally *owned* contributions but there's no notion of an equally
>> cheap local id for shared-but-not-owned contributions. Is that right?
>> 
>> I'm open to switching to Tpetra if things are better there (I may be
>> switching anyway).
>> 
>> ~ Pat
>> 
>> _______________________________________________
>> Trilinos-Users mailing list
>> Trilinos-Users at software.sandia.gov
>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>
>
>_______________________________________________
>Trilinos-Users mailing list
>Trilinos-Users at software.sandia.gov
>http://software.sandia.gov/mailman/listinfo/trilinos-users