[Trilinos-Users] Performance Question about Epetra_FECrsMatrix (and probably Tpetra::CrsMatrix too)

[Pat Notz]

Hi Chris,

Disclaimer: I don't have near the level of understanding of the
implementation issues and trade-offs that you all do. I realize I'm
speaking from a position of ignorance...

In a typical simulation, we'll re-fill a matrix 10k-100k times. So, we
spend a lot of time re-doing these tests and calculations. In general,
it would be nice to do this once and save the result. Presumably, this
would require a different flavor of SumIntoXxxValues method.

There are lots of ways one could implement this. As one example, if
the locally owned rows had local ids [0..N] then perhaps the nonlocal
contributions (exports) could have local ids [N+1..M]. Or, maybe
they're interwoven. This changes the notion of local id a bit since
ownership isn't strict but it could be much more efficient for
repeated assembly. Or, maybe this is a different kind of matrix class.

Naturally, I wouldn't expect Epetra to adopt any significant changes like this.

 ~ Pat

On Thu, Mar 31, 2011 at 8:56 AM, Baker, Christopher G. <bakercg at ornl.gov> wrote:
> Pat,
>
> The LID() lookup will happen regardless of the storage, because the row
> index passed to SumIntoGlobalValues is a global row (regardless of whether
> it is locally owned or not.)
>
> Currently, this method in Epetra_BlockMap is not inlined; neither is it
> inlined in Tpetra::Map. Perhaps it should be? That could speedup the
> computation (though measurement would be coarse, as its timing will be
> absorbed into SumIntoGlobalValues.)
>
> It is possible that this would be slower in Tpetra than Epetra, as
> Epetra_BlockMap goes through some hurdles to make this operation fast;
> Tpetra has not implemented all of these yet.
>
> Ultimately, there are two things that have to be done: determine (based on
> owned or not-owned) where/how to store the indices, and then storing the
> indices (which may entail converting the global row index to a local row).
> For non-trivial (i.e., non-contiguous) maps, even doing the latter
> requires non-trivial computational overhead.
>
> Chris
>
>
> On 3/31/11 9:32 AM, "Williams, Alan B" <william at sandia.gov> wrote:
>
>>Pat,
>>I haven't done any recent profiling of the FECrsMatrix fill operations.
>>How high are the LID operations in your profile, compared to the fill in
>>general and your coefficient-calculation code, etc.?
>>
>>I believe the same issue occurs in Tpetra, since the matrix has to decide
>>whether each contribution belongs on the local processor or not.
>>
>>Perhaps a significant change is necessary to address this. One
>>possibility would be to have the matrix not check local-vs-global for
>>each row but instead store all contributions locally and only distinguish
>>between local and non-local at the global-assemble step.
>>
>>Alan
>>
>>
>>> -----Original Message-----
>>> From: trilinos-users-bounces at software.sandia.gov [mailto:trilinos-
>>> users-bounces at software.sandia.gov] On Behalf Of Pat Notz
>>> Sent: Wednesday, March 30, 2011 12:26 PM
>>> To: trilinos-users at software.sandia.gov
>>> Subject: [Trilinos-Users] Performance Question about Epetra_FECrsMatrix
>>> (and probably Tpetra::CrsMatrix too)
>>>
>>> I'm doing typical FEM assemblies using an Epetra_FECrsMatrix and it
>>> appears to be working correctly. However, I've been profiling the
>>> serial performance and I have some concerns.
>>>
>>> In general, we're making off-processor contributions so we have to use
>>> global row/col ids. The SumIntoGlobalValues method does some
>>> gymnastics to see if the contributions are local or global and that
>>> results in a lot of calls to LID(gid) -- it's LID() that's showing up
>>> high in my profile.
>>>
>>> As far as I can tell, it appears that you can use local ids for
>>> locally *owned* contributions but there's no notion of an equally
>>> cheap local id for shared-but-not-owned contributions. Is that right?
>>>
>>> I'm open to switching to Tpetra if things are better there (I may be
>>> switching anyway).
>>>
>>> ~ Pat
>>>
>>> _______________________________________________
>>> Trilinos-Users mailing list
>>> Trilinos-Users at software.sandia.gov
>>> http://software.sandia.gov/mailman/listinfo/trilinos-users
>>
>>
>>_______________________________________________
>>Trilinos-Users mailing list
>>Trilinos-Users at software.sandia.gov
>>http://software.sandia.gov/mailman/listinfo/trilinos-users
>
>