[Trilinos-Users] Epetra_FECrsGraph and EpetraFECrsMatrix Assembly.

Wed Mar 11 04:28:05 MDT 2015

Thanks so much for the insight regarding AztecOO.
I was already using the FusedExport constructor for my second Epetra_CrsMatrix.
I think I will change my code to generate a conforming ColMap.

Are there any docs where I could have found out about these AztecOO assumptions?

Thanks again!

Kevin Tew | Assistant Professor
BYU Information Technology
265R CTB, BYU, Provo Utah 84602, USA
Phone: +1-801-422-6307
email: tew at byu.edu<mailto:tew at byu.edu>

On Mar 10, 2015, at 9:20 AM, Chris Siefert <csiefer at sandia.gov<mailto:csiefer at sandia.gov>> wrote:

Kevin,

AztecOO makes very restrictive assumptions on column maps.  This is to improve performance of the matrix-vector multiply.
Basically, all local unknowns are first, followed by remote unknowns, sorted by (owning PID, GID) pair, where "owning" is in the sense
of the domain map.  If you give Epetra a column map it will use it, regardless of whether or not it works with AztecOO.

Belos does not require the AztecOO style ordering, but  (a) it is still the default ordering for Tpetra and (b) some preconditioners may
exploit this for improved performance.

Before anything else, I would try using the "FusedImport or "FusedExport" expert routine of the  Epetra_CrsMatrix with your two matrix algorithm.
That routine does the assembly of the second matrix substantially faster than an Import / FillComplete would.

Feel free to email me off list if you have any questions.

Sincerely,
Chris Siefert

On 03/09/2015 09:39 AM, Kevin Tew wrote:
Mike and others,

I’m trying to build some adaptive IGA algorithms with smooth functions. But I’m having problems scaling to  problems.
I’m able to successfully run problems if I use the default linear maps to construct my matrix.

If I use a custom ColMap I get errors such as theses:

AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 273 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements

However if I use a the same custom ColMap and export my original assembled matrix to a second matrix built with the default linear map, the solve on the second matrix works wonderfully.
The custom ColMap hugely improves my assembly performance, but exporting to a second matrix for the solve is expensive.  I’d like to eliminate this export to a second matrix step.
I’m calling GlobalAssemble on both the FECrsGraph and FECrsMatrix with the same valid one-to-one, linear Range and Domain maps.

So my questions:
1. How can I avoid the export to a second matrix step?
2. It seems that I’m violating AztexOO’s restrictive policies on the order of column entries. Why isn’t Epetra reordering columns to meet those restrictions?
3. Would moving to Tpetra and Belos solve my problems?

Thanks,
Kevin

Small problems run just fine, but we are having problems, getting good performance out of Epetra during matrix assembly.
Nonlocal entries seem to be causing dynamic memory allocation and assembly times start to grow exponentially.
I tried constructing a Epetra_FECrsGraph with buildNonlocalGraph set to true, but when I do that my matrix solves stop converging.
I was wondering if could ask you some questions over Skype or a phone call.
I’m free today and TTHF generally.

Thanks,
Kevin

Kevin Tew | Assistant Professor
BYU Information Technology
265R CTB, BYU, Provo Utah 84602, USA
Phone: +1-801-422-6307
email: tew at byu.edu<mailto:tew at byu.edu>

On Dec 7, 2014, at 10:05 AM, Heroux, Mike <MHeroux at CSBSJU.EDU<mailto:MHeroux at CSBSJU.EDU>> wrote:

Kevin,

Generally, FECrsMatrix is used with the RowMap defined to be the same as the RangeMap, and the column map is constructed for you.  AztecOO has fairly restrictive policies on the order of column entries and Epetra reorders columns to meet those restrictions.

Is there a specific reason you want explicit control over these maps?  If so, please describe the reason.  Otherwise, I suggest you define the RowMap to be the same as the RangeMap.  Your DomainMap can be custom and different from the RangeMap.

FECrsMatrix permits you to submit entries for any row on any processor, even if those rows are not assigned to the RowMap, so you should not need to change how entries are inserted into your matrix.

Mike

From: Kevin Tew <Kevin_Tew at byu.edu<mailto:Kevin_Tew at byu.edu><mailto:Kevin_Tew at byu.edu>>
Date: Sunday, December 7, 2014 at 9:32 AM
To: "trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov><mailto:trilinos-users at software.sandia.gov>" <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov><mailto:trilinos-users at software.sandia.gov>>
Subject: [Trilinos-Users] Epetra_FECrsGraph and EpetraFECrsMatrix Assembly.

I’m trying to solve a linear system using two processors.

I’ve created an Epetra_FECrsGraph where processors have overlapping rows and cols in their Epetra_Map RowMaps and ColMaps.
I then call Epetra_FECrsGraph.GlobalAssemble( with custom Domain and Range Maps ) where the Domain and Range Maps are UnigueGID, 1-to-1 maps

I use this graph to create a EpetraFECrsMatrix, I SumIntoGlobalValues and GlobalAssemble, everything seems to go fine.

Then I attempt to use AztecOO to solve a linear system with the EpetraFECrsMatrix and I get the following error on processor 1:

AZ_extract_comm_info: Received elements must be stored after
                  all 3265 local elements

//the K matrix is globally 6531 rows by 6531 cols
//print out of K
proc_id 0: K.NumMyRows: 3855 K.NumMyCols: 3855
proc_id 1: K.NumMyRows: 3309 K.NumMyCols: 3309

// debug I added to az_comm.c
proc_id 0 - ghost 589 - cols 3266 - total 3855
proc_id 1 - ghost 387 - cols 3265 - total 3652

I’ve had difficulty finding and an example where the RowMap and ColMap are not UnigueGID, 1-to-1 maps.

Any idea what I’m doing wrong?

Thanks,

Kevin

Kevin Tew | Assistant Professor
BYU Information Technology
265R CTB, BYU, Provo Utah 84602, USA
Phone: +1-801-422-6307
email: tew at byu.edu<mailto:tew at byu.edu><mailto:tew at byu.edu>

_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.gov>
https://software.sandia.gov/mailman/listinfo/trilinos-users

_______________________________________________
Trilinos-Users mailing list
Trilinos-Users at software.sandia.gov<mailto:Trilinos-Users at software.sandia.gov>
https://software.sandia.gov/mailman/listinfo/trilinos-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20150311/718d7519/attachment.html>