[Trilinos-Users] Updating values in a Tpetra CRS matrix

Eric Marttila eric.marttila at thermoanalytics.com
Sat Sep 3 13:40:01 EDT 2016

On Friday, September 02, 2016 03:52:25 pm Hoemmen, Mark wrote:
> On 9/1/16, 5:24 PM, "Trilinos-Users on behalf of trilinos-users-
request at trilinos.org" <trilinos-users-bounces at trilinos.org on behalf 
of trilinos-users-request at trilinos.org> wrote:
> >Date: Thu, 25 Aug 2016 11:56:42 -0400
> >From: Eric Marttila <eric.marttila at thermoanalytics.com>
> >To: "trilinos-users at trilinos.org" <trilinos-users at trilinos.org>
> >Subject: [Trilinos-Users] Updating values in a Tpetra CRS matrix
> >Message-ID: <201608251156.42702.eric.marttila at thermoanalytics.com>
> >Content-Type: text/plain;  charset="us-ascii"
> >
> >I currently use an Epetra crs matrix in a newton-like solve where
> >I am repeatedly changing matrix values (but not the structure)
> >and performing linear solves.
> >
> >I'm interested in switching to Tpetra for it's multithreading
> >(OpenMP) support, but I see that the Tpetra documentation says
> >that a CRS matrix can't be changed after fillComplete() without
> >making a resumeFill() call and then another fillComplete() after
> >changing values.
> >
> >Given that fillComplete() is expensive, is it feasible for me to
> >use Tpetra in my use case?
> fillComplete is only expensive on first call, if the graph is not
> fill complete.  If the graph is fill complete,
> CrsMatrix::fillComplete is not expensive.  It costs a single
> all-reduce to figure out if anything changed on any process.  You
> can even pass fillComplete a parameter to tell it not to check,
> which makes fillComplete more or less free.
> Best practice for your use case, in both Epetra and Tpetra, is the
> following: 1. Create the CrsGraph first
> 2. Create the CrsMatrix using the constructor that takes a CrsGraph
> 3. Do your Newton solves; use sumInto*Values / replace*Values, not
> insert*Values, to change the matrix’s values
> This approach takes the least communication, performs the best, and
> has the best potential for thread parallelism.  For example,
> Tpetra::CrsMatrix’s sumInto*Values and replace*Values methods are
> thread safe and use a thread-scalable implementation.  (CrsGraph
> construction and insert*Values are not currently thread safe. 
> We’ll be working on this, but regardless, the above approach will
> always be faster.)
> Please note that not all preconditioners currently use thread
> parallelization.  If you have an interest in a particular
> preconditioner, please let us know.
> mfh

Thank you Mark, that is very helpful. In my case I am not creating a 
CrsGraph, but instead am creating a tpetra Map and then creating the 
CrsMatrix from the Map.  So my steps are:

1. Create Map
2. Create CrsMatrix from Map
3. Insert values into CrsMatrix
4. Call fillComplete
5. Peform Newton solves, and for each solve the code will:
5.0  Call resumeFill
5.1. Update values in CrsMatrix using sumInto*Values / replace*Values
         (each process only updates rows that it owns)
5.2 Call fillComplete
5.3 Call belos solve

Will the fill complete in my step 5.2 still be a cheap call as you 
describe above?  Or do I have to create the CrsGraph?

Regarding preconditioners, I am interested in using MueLu.  My 
understanding is that MueLu does use thread parallization - can you 
confirm whether or not this is true?


More information about the Trilinos-Users mailing list