[Trilinos-Users] Updating values in a Tpetra CRS matrix

[Hoemmen, Mark]

On 9/1/16, 5:24 PM, "Trilinos-Users on behalf of trilinos-users-request at trilinos.org" <trilinos-users-bounces at trilinos.org on behalf of trilinos-users-request at trilinos.org> wrote:
>Date: Thu, 25 Aug 2016 11:56:42 -0400
>From: Eric Marttila <eric.marttila at thermoanalytics.com>
>To: "trilinos-users at trilinos.org" <trilinos-users at trilinos.org>
>Subject: [Trilinos-Users] Updating values in a Tpetra CRS matrix
>Message-ID: <201608251156.42702.eric.marttila at thermoanalytics.com>
>Content-Type: text/plain;  charset="us-ascii"
>
>I currently use an Epetra crs matrix in a newton-like solve where I am 
>repeatedly changing matrix values (but not the structure) and 
>performing linear solves.
>
>I'm interested in switching to Tpetra for it's multithreading (OpenMP) 
>support, but I see that the Tpetra documentation says that a CRS 
>matrix can't be changed after fillComplete() without making a 
>resumeFill() call and then another fillComplete() after changing 
>values.
>
>Given that fillComplete() is expensive, is it feasible for me to use 
>Tpetra in my use case?

fillComplete is only expensive on first call, if the graph is not fill complete.  If the graph is fill complete, CrsMatrix::fillComplete is not expensive.  It costs a single all-reduce to figure out if anything changed on any process.  You can even pass fillComplete a parameter to tell it not to check, which makes fillComplete more or less free.

Best practice for your use case, in both Epetra and Tpetra, is the following:
1. Create the CrsGraph first
2. Create the CrsMatrix using the constructor that takes a CrsGraph
3. Do your Newton solves; use sumInto*Values / replace*Values, not insert*Values, to change the matrix’s values

This approach takes the least communication, performs the best, and has the best potential for thread parallelism.  For example, Tpetra::CrsMatrix’s sumInto*Values and replace*Values methods are thread safe and use a thread-scalable implementation.  (CrsGraph construction and insert*Values are not currently thread safe.  We’ll be working on this, but regardless, the above approach will always be faster.)

Please note that not all preconditioners currently use thread parallelization.  If you have an interest in a particular preconditioner, please let us know.

mfh