[Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix

Williams, Alan B william at sandia.gov
Tue Feb 21 09:21:17 MST 2012


Hi Eric,

Yesterday I put a fix into the trilinos votd branch to add the over-ride of the missing InsertGlobalValues method in FECrsMatrix. I believe an update will be released soon that will include that fix.

For the performance question, the extreme slow-down you are seeing is not expected.
Thanks for sending the test program, I will use that to reproduce the behavior and figure out what's going on.
There is probably a silly hot-spot in there that I can easily fix. I'll let you know what I find out.

As for other ways of doing what you want to do, you could probably do it by creating a strictly local CrsMatrix and then using import/export functionality to redistribute it. If I improve the performance of FECrsMatrix, then it should be just as good as the alternative approach, and hopefully simpler to use.

Alan


> -----Original Message-----
> From: Eric Marttila [mailto:eric.marttila at thermoanalytics.com]
> Sent: Tuesday, February 21, 2012 9:05 AM
> To: Williams, Alan B
> Cc: Littlewood, David John; 'turane at gmail.com'; 'trilinos-
> users at software.sandia.gov'; Heroux, Michael A
> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting non-local data
> in Epetra_FECrsMatrix
> 
> All,
> 
> I've been using the workaround suggested earlier (making sure I call the
> derived class version of InsertGlobalValues), and my code executes correctly
> on multiple processors.  The problem I've seen is that the insertions of non-
> local data are extremely slow.  I've attached a small bit of code which runs
> with a matrix size of 500,000.  When run with 2 processors the total insertion
> time is about 4 minutes (most of which is spent doing the non-local data).
> The insertion time seems to increase exponentially with the matrix size; a
> matrix size of 1 million takes about 20 minutes.  If the code is run with 1
> processor it runs in less than 1 second.
> 
> I saw the notes in the Epetra documentation that the GlobalAssemble() is an
> expensive process, so I was expecting somewhat of a slowdown when filling
> my
> matrix from the master process, but wasn't expecting a slowdown of this
> extent.  Also, in my case it seems that the expensive process is the
> InsertGlobalValues.  (the time for GlobalAssemble is very small in
> comparison).
> 
> I have two questions regarding this:
> 1. Is the performance that I see (with non-local insertions) the expected
> behavior for Epetra_FECrsMatrix ?
> 
> 2. Is there a different class that I should use instead of Epetra_FECrsMatrix?
> I switched from Epetra_CrsMatrix to Epetra_FECrsMatrix to get the non-local
> data functionality.  But in my case the master process is filling the entire
> matrix, so there is no chance that multiple processes would be setting values
> for a given matrix element.  I'm wondering if FECrsMatrix is overkill for my
> situation.
> 
> Thanks.
> --Eric
> 
> 
> On Wednesday, February 15, 2012 04:04:02 pm Williams, Alan B wrote:
> > Thanks Dave,
> > That may indeed be the issue. It is an oversight that only one of those
> > base-class methods is over-ridden be FECrsMatrix.
> >
> > I'm out of town this week but I'll try to get that taken care of soon after
> > I get back.
> >
> > Alan
> >
> >
> > From: Littlewood, David John
> > Sent: Wednesday, February 15, 2012 01:59 PM
> > To: Williams, Alan B; 'turane at gmail.com' <turane at gmail.com>;
> > 'eric.marttila at thermoanalytics.com' <eric.marttila at thermoanalytics.com>
> > Cc: 'trilinos-users at software.sandia.gov'
> > <trilinos-users at software.sandia.gov> Subject: Re: [Trilinos-Users]
> > [EXTERNAL] Re: Problem inserting non-local data in Epetra_FECrsMatrix
> >
> > Folks,
> >
> > I ran into a similar-sounding issue a few months ago.  I copy-and-pasted
> > the e-mail snippet below in the hope that it might help you here…
> >
> > Dave
> > --
> > David Littlewood
> > Multiphysics Simulation Technology, Dept. 1444
> > 505-284-0830
> >
> >
> > ***************************
> >
> > I've been working with Epetra_FECrsMatrix and ran into a peculiarity.  In
> > the base class Epetra_CrsMatrix, there are two functions called
> > InsertGlobalValues:
> >
> > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, const double
> > *Values, const int *Indices)
> >
> > virtual int InsertGlobalValues (int GlobalRow, int NumEntries, double
> > *Values, int *Indices)
> >
> > The only difference here is "const" on the last two arguments.
> >
> > It turns out that the first function is also implemented in the derived
> > class Epetra_FECrsMatrix, but the second one is not.  Thus if one calls
> >
> > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> > const double *Values, const int *Indices)
> >
> > you get the FE functionality (ability to insert nonlocal data), and if one
> > calls
> >
> > Epetra_FECrsMatrix::InsertGlobalValues (int GlobalRow, int NumEntries,
> > double *Values, int *Indices)
> >
> > you get the base class function, which does not handle nonlocal data.
> >
> > ***************************
> >
> >
> > From: "Williams, Alan B"
> <william at sandia.gov<mailto:william at sandia.gov>>
> > Date: Wed, 15 Feb 2012 18:55:13 +0000
> > To: "'turane at gmail.com<mailto:'turane at gmail.com>'"
> > <turane at gmail.com<mailto:turane at gmail.com>>,
> >
> "'eric.marttila at thermoanalytics.com<mailto:'eric.marttila at thermoanalytics.
> > com>'"
> >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> o
> > m>> Cc:
> > "'trilinos-users at software.sandia.gov<mailto:'trilinos-
> users at software.sandi
> > a.gov>'"
> > <trilinos-users at software.sandia.gov<mailto:trilinos-
> users at software.sandia.
> > gov>> Subject: Re: [Trilinos-Users] [EXTERNAL] Re: Problem inserting
> > non-local data in Epetra_FECrsMatrix
> >
> > Hello,
> > The usage of insertGlobalValues that Eric reported trying, should work. I
> > believe he has encountered a bug in the implementation of FECrsMatrix.
> > I'll look into that.
> >
> > Alan
> >
> >
> >
> > From: erhan turan [mailto:turane at gmail.com]
> > Sent: Wednesday, February 15, 2012 11:28 AM
> > To: Eric Marttila
> >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> o
> > m>> Cc:
> > trilinos-users at software.sandia.gov<mailto:trilinos-
> users at software.sandia.g
> > ov>
> > <trilinos-users at software.sandia.gov<mailto:trilinos-
> users at software.sandia.
> > gov>> Subject: [EXTERNAL] Re: [Trilinos-Users] Problem inserting non-local
> > data in Epetra_FECrsMatrix
> >
> > Hello,
> >
> > I had a similar problem before. My colleague Cyril Flaig found a
> > workaround. For non-local entries, you can try SumIntoGlobalValues rather
> > than InsertGlobalValues. I mean, If A.LRID()== -1 , use
> > SumIntoGlobalValues with global row index and use InsertGlobalValues
> > otherwise. That might work.
> >
> > For that purpose, what is defined was two pointers say AF and A using
> > Epetra_FECrsMatrix and Epetra_CrsMatrix, respectively. Here, we first
> > declared AF using the map and later just saying A=AF. Later all methods
> > are used for A: A->InsertGlobalValues() and A->SumIntoGlobalValues() .
> > Fill is completed using AF->GlobalAssemble(). If you try,
> > A->FillComplete() it wont fill non-local entries. That might be not the
> > best solution. Maybe there is a simpler way. But it will probably work.
> >
> > Since you mentioned this issue, I want to make a reference to
> > Epetra_FECrsMatrix documentation. There it reads:
> >
> >
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> e
> >
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4e
> 6f3ea
> > 290fe4> inserts values into the matrix only if the graph has not yet been
> > finalized
> >
> (FillComplete()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/ep
> >
> etra/doc/html/classEpetra__CrsMatrix.html#a355b4e8d1e28b16bacf37ab637
> ac0c3f
> > > has not yet been called). For non-local values, the call to
> >
> InsertGlobalValues()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> >
> es/epetra/doc/html/classEpetra__FECrsMatrix.html#a54820e91fb84c7c594c4
> e6f3e
> > a290fe4> may succeed but
> >
> theGlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packag
> e
> >
> s/epetra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b
> 86fee4
> > 59b243> method may then fail because the non-local data is not actually
> > inserted in the underlying matrix until
> >
> GlobalAssemble()<http://trilinos.sandia.gov/packages/docs/r10.8/packages/
> e
> >
> petra/doc/html/classEpetra__FECrsMatrix.html#aca5191dcdfb99eb6316b86f
> ee459b
> > 243> is called.
> >
> > This might be the reason that FillComplete() alone does not work but those
> > overloaded SumInto methods over GlobalAssemble() work. However I still
> > believe that this sentence may be vague :)
> >
> > Erhan Turan
> > Department of Computer Science
> > ETH Zurich
> >
> >
> > On Wed, Feb 15, 2012 at 4:19 PM, Eric Marttila
> >
> <eric.marttila at thermoanalytics.com<mailto:eric.marttila at thermoanalytics.c
> o
> > m>> wrote: Dear all,
> >
> > I've been using Epetra_CrsMatrix to store a linear system, but recently
> > encountered a situation where I need to fill the entire matrix from the
> > master process.  From reading the Epetra documentation I believe I need
> to
> > switch to a EpetraFECrs_Matrix to do this.
> >
> > I changed my code to use the EpetraFECrs_Matrix, but my call to
> > InsertGlobalValues() fails on the first row of the matrix that is non-local
> > to the master process.
> >
> > I've attached a small section of code that reproduces the behavior.  It
> > runs fine with 1 process, but fails when inserting row index 2 when run
> > with 2 processors.
> >
> > I would appreciate any thoughts on what I'm missing with my usage of
> > EpetraFECrs_Matrix.
> >
> > Thanks.
> > --Eric
> >
> > p.s. I'm running Trilinos 10.8.5 on linux x86_64, using mpich2.
> >
> > --
> > Eric A. Marttila
> > ThermoAnalytics, Inc.
> > 23440 Airpark Blvd.
> > Calumet, MI 49913
> >
> > email:
> >
> Eric.Marttila at ThermoAnalytics.com<mailto:Eric.Marttila at ThermoAnalytics.c
> om
> > > phone: 810-636-2443
> > fax:   906-482-9755
> > web: http://www.thermoanalytics.com
> >
> > _______________________________________________
> > Trilinos-Users mailing list
> > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> Users at software.sandia.go
> > v> http://software.sandia.gov/mailman/listinfo/trilinos-users
> >
> >
> > _______________________________________________ Trilinos-
> Users mailing list
> > Trilinos-Users at software.sandia.gov<mailto:Trilinos-
> Users at software.sandia.g
> > ov> http://software.sandia.gov/mailman/listinfo/trilinos-users
> 
> --
> Eric A. Marttila
> ThermoAnalytics, Inc.
> 23440 Airpark Blvd.
> Calumet, MI 49913
> 
> email: Eric.Marttila at ThermoAnalytics.com
> phone: 810-636-2443
> fax:   906-482-9755
> web: http://www.thermoanalytics.com



More information about the Trilinos-Users mailing list