[Trilinos-Users] Epetra_FECrsGraph and EpetraFECrsMatrix Assembly.

[Kevin Tew]

Mike and others,

I’m trying to build some adaptive IGA algorithms with smooth functions. But I’m having problems scaling to  problems.
I’m able to successfully run problems if I use the default linear maps to construct my matrix.

If I use a custom ColMap I get errors such as theses:

AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 273 local elements
AZ_extract_comm_info: Received elements must be stored after
                   all 272 local elements

However if I use a the same custom ColMap and export my original assembled matrix to a second matrix built with the default linear map, the solve on the second matrix works wonderfully.
The custom ColMap hugely improves my assembly performance, but exporting to a second matrix for the solve is expensive.  I’d like to eliminate this export to a second matrix step.
I’m calling GlobalAssemble on both the FECrsGraph and FECrsMatrix with the same valid one-to-one, linear Range and Domain maps.

So my questions:
1. How can I avoid the export to a second matrix step?
2. It seems that I’m violating AztexOO’s restrictive policies on the order of column entries. Why isn’t Epetra reordering columns to meet those restrictions?
3. Would moving to Tpetra and Belos solve my problems?

Thanks,
Kevin




Small problems run just fine, but we are having problems, getting good performance out of Epetra during matrix assembly.
Nonlocal entries seem to be causing dynamic memory allocation and assembly times start to grow exponentially.
I tried constructing a Epetra_FECrsGraph with buildNonlocalGraph set to true, but when I do that my matrix solves stop converging.
I was wondering if could ask you some questions over Skype or a phone call.
I’m free today and TTHF generally.

Thanks,
Kevin

Kevin Tew | Assistant Professor
BYU Information Technology
265R CTB, BYU, Provo Utah 84602, USA
Phone: +1-801-422-6307
email: tew at byu.edu<mailto:tew at byu.edu>

On Dec 7, 2014, at 10:05 AM, Heroux, Mike <MHeroux at CSBSJU.EDU<mailto:MHeroux at CSBSJU.EDU>> wrote:

Kevin,

Generally, FECrsMatrix is used with the RowMap defined to be the same as the RangeMap, and the column map is constructed for you.  AztecOO has fairly restrictive policies on the order of column entries and Epetra reorders columns to meet those restrictions.

Is there a specific reason you want explicit control over these maps?  If so, please describe the reason.  Otherwise, I suggest you define the RowMap to be the same as the RangeMap.  Your DomainMap can be custom and different from the RangeMap.

FECrsMatrix permits you to submit entries for any row on any processor, even if those rows are not assigned to the RowMap, so you should not need to change how entries are inserted into your matrix.

Mike

From: Kevin Tew <Kevin_Tew at byu.edu<mailto:Kevin_Tew at byu.edu><mailto:Kevin_Tew at byu.edu>>
Date: Sunday, December 7, 2014 at 9:32 AM
To: "trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov><mailto:trilinos-users at software.sandia.gov>" <trilinos-users at software.sandia.gov<mailto:trilinos-users at software.sandia.gov><mailto:trilinos-users at software.sandia.gov>>
Subject: [Trilinos-Users] Epetra_FECrsGraph and EpetraFECrsMatrix Assembly.

I’m trying to solve a linear system using two processors.

I’ve created an Epetra_FECrsGraph where processors have overlapping rows and cols in their Epetra_Map RowMaps and ColMaps.
I then call Epetra_FECrsGraph.GlobalAssemble( with custom Domain and Range Maps ) where the Domain and Range Maps are UnigueGID, 1-to-1 maps

I use this graph to create a EpetraFECrsMatrix, I SumIntoGlobalValues and GlobalAssemble, everything seems to go fine.

Then I attempt to use AztecOO to solve a linear system with the EpetraFECrsMatrix and I get the following error on processor 1:

AZ_extract_comm_info: Received elements must be stored after
                  all 3265 local elements

//the K matrix is globally 6531 rows by 6531 cols
//print out of K
proc_id 0: K.NumMyRows: 3855 K.NumMyCols: 3855
proc_id 1: K.NumMyRows: 3309 K.NumMyCols: 3309

// debug I added to az_comm.c
proc_id 0 - ghost 589 - cols 3266 - total 3855
proc_id 1 - ghost 387 - cols 3265 - total 3652

I’ve had difficulty finding and an example where the RowMap and ColMap are not UnigueGID, 1-to-1 maps.

Any idea what I’m doing wrong?

Thanks,

Kevin

Kevin Tew | Assistant Professor
BYU Information Technology
265R CTB, BYU, Provo Utah 84602, USA
Phone: +1-801-422-6307
email: tew at byu.edu<mailto:tew at byu.edu><mailto:tew at byu.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://software.sandia.gov/pipermail/trilinos-users/attachments/20150309/e29b88e2/attachment-0001.html>